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Preface 


This is a free textbook for an undergraduate course on the Theory of Com¬ 
putation, which we have been teaching at Carleton University since 2002. 
Until the 2011/2012 academic year, this course was offered as a second-year 
course (COMP 2805) and was compulsory for all Computer Science students. 
Starting with the 2012/2013 academic year, the course has been downgraded 
to a third-year optional course (COMP 3803). 

We have been developing this book since we started teaching this course. 
Currently, we cover most of the material from Chapters 2-5 during a 12-week 
term with three hours of classes per week. 

The material from Chapter 6, on Complexity Theory, is taught in the 
third-year course COMP 3804 (Design and Analysis of Algorithms). In the 
early years of COMP 2805, we gave a two-lecture overview of Complexity 
Theory at the end of the term. Even though this overview has disappeared 
from the course, we decided to keep Chapter 6. This chapter has not been 
revised/modified for a long time. 

The course as we teach it today has been influenced by the following two 
textbooks: 

• Introduction to the Theory of Computation (second edition), by Michael 
Sipser, Thomson Course Technnology, Boston, 2006. 

• Einfuhrung in die Theoretische Informatik, by Klaus Wagner, Springer- 
Verlag, Berlin, 1994. 

Besides reading this text, we recommend that you also take a look at 
these excellent textbooks, as well as one or more of the following ones: 

• Elements of the Theory of Computation (second edition), by Harry 
Lewis and Christos Papadimitriou, Prentice-Hall, 1998. 
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• Introduction to Languages and the Theory of Computation (third edi¬ 
tion), by John Martin, McGraw-Hill, 2003. 

• Introduction to Automata Theory, Languages, and Computation (third 
edition), by John Hopcroft, Raieev Motwani, Jeffrey Ullman, Addison 
Wesley, 2007. 

Please let us know if you find errors, typos, simpler proofs, comments, 
omissions, or if you think that some parts of the book “need improvement”. 


| You are free to copy, distribute, transmit, and adapt this textbook. 
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Chapter 1 
Introduction 


1.1 Purpose and motivation 

This course is on the Theory of Computation, which tries to answer the 
following questions: 

• What are the mathematical properties of computer hardware and soft¬ 
ware? 

• What is a computation and what is an algorithm ? Can we give rigorous 
mathematical definitions of these notions? 

• What are the limitations of computers? Can “everything” be com¬ 
puted? (As we will see, the answer to this question is “no”.) 


Purpose of the Theory of Computation: Develop formal math¬ 
ematical models of computation that reflect real-world computers. 


This field of research was started by mathematicians and logicians in the 
1930’s, when they were trying to understand the meaning of a “computation”. 
A central question asked was whether all mathematical problems can be 
solved in a systematic way. The research that started in those days led to 
computers as we know them today. 

Nowadays, the Theory of Computation can be divided into the follow¬ 
ing three areas: Complexity Theory, Computability Theory, and Automata 
Theory. 
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1.1.1 Complexity theory 

The main question asked in this area is “What makes some problems com¬ 
putationally hard and other problems easy?" 

Informally, a problem is called “easy”, if it is efficiently solvable. Exam¬ 
ples of “easy” problems are (i) sorting a sequence of, say, 1,000,000 numbers, 
(ii) searching for a name in a telephone directory, and (iii) computing the 
fastest way to drive from Ottawa to Miami. On the other hand, a problem is 
called “hard”, if it cannot be solved efficiently, or if we don’t know whether 
it can be solved efficiently. Examples of “hard” problems are (i) time table 
scheduling for all courses at Carleton, (ii) factoring a 300-digit integer into 
its prime factors, and (iii) computing a layout for chips in VLSI. 

Central Question in Complexity Theory: Classify problems ac¬ 
cording to their degree of “difficulty”. Give a rigorous proof that 
problems that seem to be “hard” are really “hard”. 


1.1.2 Computability theory 

In the 1930’s, Godel, Turing, and Church discovered that some of the fun¬ 
damental mathematical problems cannot be solved by a “computer”. (This 
may sound strange, because computers were invented only in the 1940’s). 
An example of such a problem is “Is an arbitrary mathematical statement 
true or false?” To attack such a problem, we need formal definitions of the 
notions of 

• computer, 

• algorithm, and 

• computation. 

The theoretical models that were proposed in order to understand solvable 
and unsolvable problems led to the development of real computers. 

Central Question in Computability Theory: Classify problems | 

as being solvable or unsolvable._I 


Source: www.csitsansar.com 







1.1. Purpose and motivation 


3 


1.1.3 Automata theory 

Automata Theory deals with definitions and properties of different types of 
“computation models”. Examples of such models are: 

• Finite Automata. These are used in text processing, compilers, and 
hardware design. 

• Context-Free Grammars. These are used to define progra mming lan¬ 
guages and in Artificial Intelligence. 

• Turing Machines. These form a simple abstract model of a “real” 
computer, such as your PC at home. 


Central Question in Automata Theory: Do these models have 
the same power, or can one model solve more problems than the 
other? _ 


1.1.4 This course 

In this course, we will study the last two areas in reverse order: We will start 
with Automata Theory, followed by Computability Theory. The first area, 
Complexity Theory, will be covered in COMP 3804. 

Actually, before we start, we will review some mathematical proof tech¬ 
niques. As you may guess, this is a fairly theoretical course, with lots of 
definitions, theorems, and proofs. You may guess this course is fun stuff for 
math lovers, but boring and irrelevant for others. You guessed it wrong, and 
here are the reasons: 

1. This course is about the fundamental capabilities and limitations of 
computers. These topics form the core of computer science. 

2. It is about mathematical properties of computer hardware and software. 

3. This theory is very much relevant to practice, for example, in the design 
of new programming languages, compilers, string searching, pattern 
matching, computer security, artificial intelligence, etc., etc. 

4. This course helps you to learn problem solving skills. Theory teaches 
you how to think, prove, argue, solve problems, express, and abstract. 
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5. This theory simplifies the complex computers to an abstract and simple 
mathematical model, and helps you to understand them better. 

6. This course is about rigorously analyzing capabilities and limitations 
of systems. 

Where does this course fit in the Computer Science Curriculum at Car- 
leton University? It is a theory course that is the third part in the series 
COMP 1805, COMP 2804, COMP 3803, COMP 3804, and COMP 4804. 
This course also widens your understanding of computers and will influence 
other courses including Compilers, Programming Languages, and Artificial 
Intelligence. 


1.2 Mathematical preliminaries 

Throughout this course, we will assume that you know the following mathe¬ 
matical concepts: 

1. A set is a collection of well-defined objects. Examples are (i) the set of 
all Dutch Olympic Gold Medallists, (ii) the set of all pubs in Ottawa, 
and (iii) the set of all even natural numbers. 

2. The set of natural numbers is N = {1, 2,3,...}. 

3. The set of integers is Z = {..., —3, —2, —1, 0,1, 2, 3,...}. 

4. The set of rational numbers is Q = { m/n :m6Z,neZ,fi^0}. 

5. The set of real numbers is denoted by 1R. 

6. If A and B are sets, then A is a subset of B, written as AC B, if every 
element of A is also an element of B. For example, the set of even 
natural numbers is a subset of the set of all natural numbers. Every 
set A is a subset of itself, i.e., A C A. The empty set is a subset of 
every set A, i.e., 0 C A. 

7. If B is a set, then the power set V(B) of B is defined to be the set of 
all subsets of B: 

V{B) = {A : A C B}. 

Observe that 0 e V(B) and B e V(B). 
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8. If A and B are two sets, then 

(a) their union is defined as 

AU B = {x : x e A or x e B }, 

(b) their intersection is defined as 

AnB = {x : x e A and x € B }, 

(c) their difference is defined as 

A \ B — {x : x e A and x $ B }, 

(d) the Cartesian product of A and B is defined as 

Ax B = {(x, y) : x € A and y € B}, 

(e) the complement of A is defined as 

A = {x : x £ A}. 

9. A binary relation on two sets A and B is a subset of Ax B. 

10. A function f from A to B, denoted by / : A —y B, is a binary relation 
R , having the property that for each element a e A, there is exactly 
one ordered pair in R, whose first component is a. We will also say 
that f(a ) — b, or f maps a to b, or the image of a under / is b. The 
set A is called the domain of /, and the set 

{b G B : there is an a e A with /(a) = b} 
is called the range of /. 

11. A function / : A —>• B is one-to-one (or injective ), if for any two distinct 
elements a and a' in A, we have f(a ) f f(o r ). The function / is onto 
(or surjective ), if for each element b 6 B, there exists an element a G A, 
such that f(a) = 6; in other words, the range of / is equal to the set 
B. A function / is a bijection, if / is both injective and surjective. 

12. A binary relation R C A x A is an equivalence relation , if it satisfies 
the following three conditions: 


Source: www.csitsansar.com 



6 


Chapter 1. Introduction 


(a) R is reflexive: For every element in a E A, we have (a, a) E R. 

(b) R is symmetric: For all a and b in A, if (a, b ) E R, then also 
( b, a) E R. 

(c) R is transitive: For all a, b, and c in A, if (a, b) E R and ( b, c) E R, 
then also (a, c ) E R. 

13. A graph G = (V, E ) is a pair consisting of a set V, whose elements are 
called vertices, and a set E, where each element of E is a pair of distinct 
vertices. The elements of E are called edges. The figure below shows 
some well-known graphs: K 5 (the complete graph on five vertices), K 3>3 
(the complete bipartite graph on 2 x 3 = 6 vertices), and the Peterson 
graph. 



Peterson graph 


The degree of a vertex v, denoted by deg{y), is defined to be the number 
of edges that are incident on v. 

A path in a graph is a sequence of vertices that are connected by edges. 
A path is a cycle, if it starts and ends at the same vertex. A simple 
path is a path without any repeated vertices. A graph is connected , if 
there is a path between every pair of vertices. 

14. In the context of strings, an alphabet is a finite set, whose elements 
are called symbols. Examples of alphabets are E = {0,1} and E = 
{a,b, c, ...,z}. 

15. A string over an alphabet E is a finite sequence of symbols, where each 
symbol is an element of E. The length of a string w , denoted by |w|, is 
the number of symbols contained in w. The empty string, denoted by 
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e, is the string having length zero. For example, if the alphabet E is 
equal to {0,1}, then 10, 1000, 0, 101, and e are strings over E, having 
lengths 2, 4, 1, 3, and 0, respectively. 

16. A language is a set of strings. 

17. The Boolean values are 1 and 0, that represent true and false, respec¬ 
tively. The basic Boolean operations include 

(a) negation (or NOT), represented by -i, 

(b) conjunction (or AND), represented by A, 

(c) disjunction (or OR), represented by V, 

(d) exclusive-or (or XOR), represented by ©, 

(e) equivalence, represented by or <=>, 

(f) implication, represented by -4 or =X 


The following table explains the meanings of these operations. 


NOT 

AND 

OR 

XOR 

equivalence 

implication 

i0 = 1 

0 A0 = 0 

OVO = 0 

0® 0 = 0 

0f)0=l 

0^0 = 1 

-.1 = 0 

0 A 1 = 0 

0V1 = 1 

0®1 = 1 

0^1 = 0 

0-41 = 1 


O 

II 

o 

< 

1 V0 = 1 

1®0 = 1 

lf)0 = 0 

1^0 = 0 


1 A1 = 1 

1 VI = 1 

101 = 0 

1^1 = 1 

1-41 = 1 


1.3 Proof techniques 

In mathematics, a theorem is a statement that is true. A proof is a sequence 
of mathematical statements that form an argument to show that a theorem is 
true. The statements in the proof of a theorem include axioms (assumptions 
about the underlying mathematical structures), hypotheses of the theorem 
to be proved, and previously proved theorems. The main question is “How 
do we go about proving theorems?” This question is similar to the question 
of how to solve a given problem. Of course, the answer is that finding proofs, 
or solving problems, is not easy; otherwise life would be dull! There is no 
specified way of coming up with a proof, but there are some generic strategies 
that could be of help. In this section, we review some of these strategies, 
that will be sufficient for this course. The best way to get a feeling of how 
to come up with a proof is by solving a large number of problems. Here are 
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some useful tips. (You may take a look at the book How to Solve It, by G. 
Polya). 

1. Read and completely understand the statement of the theorem to be 
proved. Most often this is the hardest part. 

2. Sometimes, theorems contain theorems inside them. For example, 
“Property A if and only if property B" , requires showing two state¬ 
ments: 

(a) If property A is true, then property B is true (A =>• B). 

(b) If property B is true, then property A is true (B =>• A). 

Another example is the theorem “Set A equals set B .” To prove this, 
we need to prove that A C B and B C A. That is, we need to show 
that each element of set A is in set B, and that each element of set B 
is in set A. 

3. Try to work out a few simple cases of the theorem just to get a grip on 
it (i.e., crack a few simple cases first). 

4. Try to write down the proof once you have it. This is to ensure the 
correctness of your proof. Often, mistakes are found at the time of 
writing. 

5. Finding proofs takes time, we do not come prewired to produce proofs. 
Be patient, think, express and write clearly and try to be precise as 
much as possible. 

In the next sections, we will go through some of the proof strategies. 

1.3.1 Direct proofs 

As the name suggests, in a direct proof of a theorem, we just approach the 
theorem directly. 

Theorem 1.3.1 If n is an odd positive integer, then n 2 is odd as well. 
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Proof. An odd positive integer n can be written as n = 2k + 1, for some 
integer k > 0. Then 

n 2 = (2k + l) 2 - 4A; 2 + + 1 = 2(2fc 2 + 2fc) + 1. 

Since 2(2A: 2 + 2k) is even, and “even plus one is odd”, we can conclude that 
n 2 is odd. ■ 


Theorem 1.3.2 Let G = (V, E) be a graph. Then the sum of the degrees of 
all vertices is an even integer, i.e., 

de 9^ 

v&V 

is even. 

Proof. If you do not see the meaning of this statement, then first try it out 
for a few graphs. The reason why the statement holds is very simple: Each 
edge contributes 2 to the summation (because an edge is incident on exactly 
two distinct vertices). ■ 

Actually, the proof above proves the following theorem. 

Theorem 1.3.3 Let G = (V, E) be a graph. Then the sum of the degrees of 
all vertices is equal to twice the number of edges, i.e., 

Y,deg(v)=2\E\. 

vev 

1.3.2 Constructive proofs 

This technique not only shows the existence of a certain object, it actually 
gives a method of creating it. Here is how a constructive proof looks like: 

Theorem 1.3.4 There exists an object with property V. 

Proof. Here is the object: [...] 

And here is the proof that the object satisfies property V: [...] ■ 

Here is an example of a constructive proof. A graph is called 3- regular, if 
each vertex has degree three. 
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Theorem 1.3.5 For every even integer n > 4, there exists a 3-regular graph 
with n vertices. 

Proof. Define 

{0,1,2,..., n-1}, 

and 

E = {{i,i+l} : 0 < i < n— 2}U{{n — 1, 0}}U{{i, i+n/2} : 0 <i< n/ 2 — 1}. 

Then the graph G = (V, E ) is 3-regular. 

Convince yourself that this graph is indeed 3-regular. It may help to draw 
the graph for, say, n = 8. ■ 


1.3.3 Nonconstructive proofs 

In a nonconstructive proof, we show that a certain object exists, without 
actually creating it. Here is an example of such a proof: 

Theorem 1.3.6 There exist irrational numbers x and y such that x y is ra¬ 
tional. 

Proof. There are two possible cases. 

Case 1: y/2^ e Q. 

In this case, we take x = y = \[2. In Theorem 1.3.9 below, we will prove 
that \f2 is irrational. 

Case 2: & Q- 

In this case, we take x = y/2^ and y = y/2. Since 
x y = = sfi = 2, 

the claim in the theorem follows. ■ 

Observe that this proof indeed proves the theorem, but it does not give 
an example of a pair of irrational numbers x and y such that x y is rational. 
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1.3.4 Proofs by contradiction 

This is how a proof by contradiction looks like: 

Theorem 1.3.7 Statement S is true. 

Proof. Assume that statement S is false. Then, derive a contradiction (such 
as 1 + 1 = 3). 

In other words, show that the statement U S => false ” is true. This is 
sufficient, because the contrapositive of the statement “S => false" is the 
statement “true =>- S". The latter logical formula is equivalent to S, and 
that is what we wanted to show. ■ 

Below, we give two examples of proofs by contradiction. 

Theorem 1.3.8 Let n be a positive integer. If n 2 is even, then n is even. 

Proof. We will prove the theorem by contradiction. So we assume that n 2 
is even, but n is odd. Since n is odd, we know from Theorem 1.3.1 that n 2 
is odd. This is a contradiction, because we assumed that n 2 is even. ■ 


Theorem 1.3.9 \[2 is irrational, i.e., \/2 cannot be written as a fraction of 
two integers m and n. 

Proof. We will prove the theorem by contradiction. So we assume that y/2 
is rational. Then \f2 can be written as a fraction of two integers, \/2 = m/n, 
where m > 1 and n > 1. We may assume that m and n are not both 
even, because otherwise we can get rid of the common factors. By squaring 
\/2 = m/n , we get 2 n 2 — m 2 . This implies that m 2 is even. Then, by 
Theorem 1.3.8, m is even, which means that we can write m as m = 2 k, for 
some positive integer k. It follows that 2 n 2 — m 2 = 4 k 2 , which implies that 
n 2 = 2k 2 . Hence, n 2 is even. Again by theorem 1.3.8, it follows that n is 
even. 

We have shown that m and n are both even. But we know that m and 
n are not both even. Hence, we have a contradiction. Our assumption that 
\/2 is rational is wrong. Thus, we can conclude that \/2 is irrational. ■ 

There is a nice discussion of this proof in the book My Brain is Open: 
The Mathematical Journeys of Paul Erdos by B. Schechter. 
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1.3.5 The pigeon hole principle 

This is a simple principle with surprising consequences. 

Pigeon Hole Principle: If n + 1 or more objects are placed into n 
boxes, then there is at least one box containing two or more objects. 
In other words, if A and B are two sets such that \A\ > \B\, then 
there is no one-to-one function from A to B. _ 


Theorem 1.3.10 Let n be a positive integer. Every sequence of n 2 + 1 dis¬ 
tinct real numbers contains a subsequence of length n + 1 that is either in¬ 
creasing or decreasing. 

Proof. For example consider the sequence (20,10, 9, 7,11, 2, 21,1, 20, 31) of 
10 = 3 2 + 1 numbers. This sequence contains an increasing subsequence of 
length 4 = 3 + 1, namely (10,11, 21, 31). 

The proof of this theorem is by contradiction, and uses the pigeon hole 
principle. 

Let (ai,a 2 ,... ,a n 2 +1 ) be an arbitrary sequence of n 2 + 1 distinct real 
numbers. For each i with 1 < i < n 2 + 1, let inc l denote the length of 
the longest increasing subsequence that starts at a ;, and let dec t denote the 
length of the longest decreasing subsequence that starts at cq. 

Using this notation, the claim in the theorem can be formulated as follows: 
There is an index i such that me* >n + 1 or dcc; L > n + 1. 

We will prove the claim by contradiction. So we assume that me* < n 
and deci < n for all i with 1 < i < n 2 + 1. 

Consider the set 


B = {(6, c) : 1 < b < n, 1 < c < n}, 

and think of the elements of B as being boxes. For each i with 1 < i < n 2 + l, 
the pair (me,, deef) is an element of B. So we have n 2 +1 elements (me*, deef), 
which are placed in the n 2 boxes of B. By the pigeon hole principle, there 
must be a box that contains two (or more) elements. In other words, there 
exist two integers i and j such that i < j and 

( inci , deci ) = ( incj , decj). 

Recall that the elements in the sequence are distinct. Hence, a x / aj. We 
consider two cases. 
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First assume that a t < dj. Then the length of the longest increasing 
subsequence starting at a* must be at least 1 + inCj, because we can append a, : 
to the longest increasing subsequence starting at a,-. Therefore, inc% ^ incj, 
which is a contradiction. 

The second case is when a* > a r Then the length of the longest decreasing 
subsequence starting at a t must be at least 1 + dec 3 , because we can append a* 
to the longest decreasing subsequence starting at a 3 . Therefore, dec t ^ dec 3 , 
which is again a contradiction. ■ 


1.3.6 Proofs by induction 

This is a very powerful and important technique for proving theorems. 

For each positive integer n , let P(n) be a mathematical statement that 
depends on n. Assume we wish to prove that P(n) is true for all positive 
integers n. A proof by induction of such a statement is carried out as follows: 

Basis: Prove that P(l) is true. 

Induction step: Prove that for all n > 1, the following holds: If P{n) is 
true, then P(n + 1) is also true. 

In the induction step, we choose an arbitrary integer n > 1 and assume 
that P(n) is true; this is called the induction hypothesis. Then we prove that 
P(n+ 1) is also true. 


Theorem 1.3.11 For all positive integers n, we have 


1 + 2 + 3+ .. .+77, 


n(n + 1) 
2 


Proof. We start with the basis of the induction. If n = 1, then the left-hand 
side is equal to 1, and so is the right-hand side. So the theorem is true for 
n = l. 

For the induction step, let n > 1 and assume that the theorem is true for 
7i, i.e., assume that 


1 + 2 + 3+ ... + 7i 


n(n + 1) 
2 
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We have to prove that the theorem is true for n + 1, i.e., we have to prove 
that 


l + 2 + 3 + ... + (n + l) — 


(n+ l)(n + 2) 


Here is the proof: 

l + 2 + 3 + ... + (n + l) = 1 4- 2 + 3 + ... + + 1) 


n(n + 1) 


+ (n + 1) 


(n + l)(n + 2) 


By the way, here is an alternative proof of the theorem above: Let S — 
1 + 2 + 3+ ... + 7i. Then, 



Since there are n terms on the right-hand side, we have 25' = n(n + 1). This 
implies that 5 = n{n + l)/2. 

Theorem 1.3.12 For every positive integer n, a — b is a factor of a n — b n . 
Proof. A direct proof can be given by providing a factorization of a n — b n : 
a n — b n — (a — &)(a n_1 + a n ~ 2 b + a n ~ 3 b 2 + ,*,., + ab n ~ 2 + b n ~ l ). 

We now prove the theorem by induction. For the basis, let n = 1. The claim 
in the theorem is “a — b is a factor of a — b ”, which is obviously true. 

Let n > 1 and assume that a —b is a factor of a n — b n . We have to prove 
that a — b is a factor of a n+1 — b n+1 . We have 

a n+l _ b n+l = a n +1 _ ^ + ^ _ fc n+l = a n^ _ + ( fl n _ ^ 

The first term on the right-hand side is divisible by a — b. By the induction 
hypothesis, the second term on the right-hand side is divisible by a — b as 
well. Therefore, the entire right-hand side is divisible by a — b. Since the 
right-hand side is equal to a n+1 — b n+1 , it follows that a — b is a factor of 

a n+i _ h n +1 B 

We now give an alternative proof of Theorem 1.3.3: 
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Theorem 1.3.13 Let G = (V, E) be a graph with m edges. Then the sum 
of the degrees of all vertices is equal to twice the number of edges, i.e., 

^2 de 9( v ) = 2m. 

vGV 

Proof. The proof is by induction on the number m of edges. For the basis of 
the induction, assume that m = 0. Then the graph G does not contain any 
edges and, therefore, Y^veV de fj( v ) — 0. Thus, the theorem is true if m = 0. 

Let m > 0 and assume that the theorem is true for every graph with m 
edges. Let G be an arbitrary graph with m +1 edges. We have to prove that 

J2vev de 9( v ) = 2 ( m + !)■ 

Let {a, b} be an arbitrary edge in G, and let G' be the graph obtained 
from G by removing the edge {a, b}. Since G' has m edges, we know from 
the induction hypothesis that the sum of the degrees of all vertices in G' is 
equal to 2m. Using this, we obtain 

y] deg(v) — ^2 de 9( v ) + 2 = 2m + 2 = 2(m + 1). 

v&G v&G' 


1.3.7 More examples of proofs 

Recall Theorem 1.3.5, which states that for every even integer n > 4, there 
exists a 3-regular graph with n vertices. The following theorem explains why 
we stated this theorem for even values of n. 

Theorem 1.3.14 Let n > 5 be an odd integer. There is no 3-regular graph 
with n vertices. 

Proof. The proof is by contradiction. So we assume that there exists a 
graph G = (V, E) with n vertices that is 3-regular. Let m be the number of 
edges in G. Since deg(v) = 3 for every vertex, we have 

y deg(v) — 3 n. 

vGV 

On the other hand, by Theorem 1.3.3, we have 
y: deg{v) — 2m. 

vGV 
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It follows that 3 n = 2m, which can be rewritten as nn = 3n/2. Since m is an 
integer, and since gcd( 2,3) = 1, n/2 must be an integer. Hence, n is even, 
which is a contradiction. I 

Let K n be the complete graph on n vertices. This graph has a vertex set 
of size n, and every pair of distinct vertices is joined by an edge. 

If G — (V, E) is a graph with n vertices, then the complement G of G is 
the graph with vertex set V that consists of those edges of K n that are not 
present in G. 

Theorem 1.3.15 Let n > 2 and let G be a graph on n vertices. Then G is 
connected or G is connected. 

Proof. We prove the theorem by induction on the number n of vertices. For 
the basis, assume that n = 2. There are two possibilities for the graph G: 

1. G contains one edge. In this case, G is connected. 

2. G does not contain an edge. In this case, the complement G contains 
one edge and, therefore, G is connected. 

So for n = 2, the theorem is true. 

Let n > 2 and assume that the theorem is true for every graph with n 
vertices. Let G be graph with n + 1 vertices. We have to prove that G is 
connected or G is connected. We consider three cases. 

Case 1: There is a vertex v whose degree in G is equal to n. 

Since G has n +1 vertices, v is connected by an edge to every other vertex 
of G. Therefore, G is connected. 

Case 2: There is a vertex v whose degree in G is equal to 0. 

In this case, the degree of v in the graph G is equal to n. Since G has n + l 
vertices, v is connected by an edge to every other vertex of G. Therefore, G 
is connected. 

Case 3: For every vertex v, the degree of v in G is in {1,2,..., n — 1}. 

Let v be an arbitrary vertex of G. Let G' be the graph obtained by 
deleting from G the vertex v, together with all edges that are incident on v. 
Since G' has n vertices, we know from the induction hypothesis that G' is 
connected or G' is connected. 
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Let us first assume that G' is connected. Then the graph G is connected 
as well, because there is at least one edge in G between v and some vertex 
of G'. 

If G' is not connected, then G' must be connected. Since we are in Case 3, 
we know that the degree of v in G is in the set {1, 2,...,n — 1}. It follows 
that the degree of v in the graph G is in this set as well. Hence, there is at 
least one edge in G between v and some vertex in G'. This implies that G is 
connected. ■ 

The previous theorem can be rephrased as follows: 

Theorem 1.3.16 Let n > 2 and consider the complete graph K n on n ver¬ 
tices. Color each edge of this graph as either red or blue. Let R be the graph 
consisting of all the red edges, and let B be the graph consisting of all the 
blue edges. Then R is connected or B is connected. 

A graph is said to be planar , if it can be drawn (a better term is “embed¬ 
ded”) in the plane in such a way that no two edges intersect, except possibly 
at their endpoints. An embedding of a planar graph consists of vertices, 
edges, and faces. In the example below, there are 11 vertices, 18 edges, and 
9 faces (including the unbounded face). 



The following theorem is known as Euler’s theorem for planar graphs. 
Apparently, this theorem was discovered by Euler around 1750. Legendre 
gave the first proof in 1794, see 

http://www.ics.uci.edu/~eppstein/junkyard/euler/ 

Theorem 1.3.17 (Euler) Consider an embedding of a planar graph G. Let 
v, e, and f be the number of vertices, edges, and faces (including the single 
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unbounded face) of this embedding, respectively. Moreover, letc be the number 
of connected components of G. Then 

v ~e + f = c+ 1 . 

Proof. The proof is by induction on the number of edges of G. To be more 
precise, we start with a graph having no edges, and prove that the theorem 
holds for this case. Then, we add the edges one by one, and show that the 
relation v — e + f = c+ lis maintained. 

So we first assume that G has no edges, i.e., e = 0. Then the embedding 
consists of a collection of v points. In this case, we have / = 1 and c — v. 
Hence, the relation v — e + f — c+ 1 holds. 

Let e > 0 and assume that Euler’s formula holds for a subgraph of G 
having e — 1 edges. Let { u , v} be an edge of G that is not in the subgraph, 
and add this edge to the subgraph. There are two cases depending on whether 
this new edge joins two connected components or joins two vertices in the 
same connected component. 

Case 1: The new edge {u,v} joins two connected components. 

In this case, the number of vertices and the number of faces do not change, 
the number of connected components goes down by 1, and the number of 
edges increases by 1. It follows that the relation in the theorem is still valid. 

Case 2: The new edge {u, v} joins two vertices in the same connected com¬ 
ponent. 

In this case, the number of vertices and the number of connected com¬ 
ponents do not change, the number of edges increases by 1, and the number 
of faces increases by 1 (because the new edge splits one face into two faces). 
Therefore, the relation in the theorem is still valid. ■ 

Euler’s theorem is usually stated as follows: 

Theorem 1.3.18 (Euler) Consider an embedding of a connected planar 
graph G. Let v, e, and f be the number of vertices, edges, and faces (in¬ 
cluding the single unbounded face) of this embedding, respectively. Then 

v — e + f = 2. 

If you like surprising proofs of various mathematical results, you should 
read the book Proofs from THE BOOK by Aigner and Ziegler. 
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Exercises 

1.1 Use induction to prove that every positive integer can be written as a 
product of prime numbers. 

1.2 For every prime number p, prove that y/p is irrational. 

1.3 Let n be a positive integer that is not a perfect square. Prove that y/n 
is irrational. 

1.4 Prove by induction that n 4 — 4 n 2 is divisible by 3, for all integers n > 1. 

1.5 Prove that 

E -i < 2 - V", 

for every integer n > 2. 

1.6 Prove that 9 divides n 3 + (n + l) 3 + (n + 2) 3 , for every integer n > 0. 

1.7 Prove that in any set of n + 1 numbers from {1, 2,..., 2n}, there are 
always two numbers that are consecutive. 

1.8 Prove that in any set of n + 1 numbers from {1, 2,..., 2 n}, there are 
always two numbers such that one divides the other. 
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Chapter 2 

Finite Automata and Regular 
Languages 


In this chapter, we introduce and analyze the class of languages that are 
known as regular languages. Informally, these languages can be “processed” 
by computers having a very small amount of memory. 


2.1 An example: Controling a toll gate 

Before we give a formal definition of a finite automaton, we consider an 
example in which such an automaton shows up in a natural way We consider 
the problem of designing a “computer” that controls a toll gate. 

When a car arrives at the toll gate, the gate is closed. The gate opens as 
soon as the driver has payed 25 cents. We assume that we have only three 
coin denominations: 5, 10, and 25 cents. We also assume that no excess 
change is returned. 

After having arrived at the toll gate, the driver inserts a sequence of coins 
into the machine. At any moment, the machine has to decide whether or not 
to open the gate, i.e., whether or not the driver has paid 25 cents (or more). 
In order to decide this, the machine is in one of the following six states, at 
any moment during the process: 

• The machine is in state q 0 , if it has not collected any money yet. 

• The machine is in state q\, if it has collected exactly 5 cents. 

• The machine is in state q 2 , if it has collected exactly 10 cents. 
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• The machine is in state g 3 , if it has collected exactly 15 cents. 

• The machine is in state g 4 , if it has collected exactly 20 cents. 

• The machine is in state g 5 , if it has collected 25 cents or more. 

Initially (when a car arrives at the toll gate), the machine is in state g 0 . 

Assume, for example, that the driver presents the sequence (10,5,5,10) of 
coins. 


• After receiving the first 10 cents coin, the machine switches from state 
go to state g 2 . 

• After receiving the first 5 cents coin, the machine switches from state 
g 2 to state g 3 . 

• After receiving the second 5 cents coin, the machine switches from state 
g 3 to state g 4 . 

• After receiving the second 10 cents coin, the machine switches from 
state g 4 to state g 5 . At this moment, the gate opens. (Remember that 
no change is given.) 

The figure below represents the behavior of the machine for all possible 
sequences of coins. State q 5 is represented by two circles, because it is a 
special state: As soon as the machine reaches this state, the gate opens. 



Observe that the machine (or computer) only has to remember which 
state it is in at any given time. Thus, it needs only a very small amount 
of memory: It has to be able to distinguish between any one of six possible 
cases and, therefore, it only needs a memory of [log 6] = 3 bits. 
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2.2 Deterministic finite automata 

Let us look at another example. Consider the following state diagram : 



0,1 


We say that q\ is the start state and q 2 is an accept state. Consider the 
input string 1101. This string is processed in the following way: 

• Initially, the machine is in the start state q\. 

• After having read the first 1, the machine switches from state q\ to 
state q 2 . 

• After having read the second 1, the machine switches from state q 2 to 
state q 2 . (So actually, it does not switch.) 

• After having read the first 0, the machine switches from state q 2 to 
state q 3 . 

• After having read the third 1, the machine switches from state q 3 to 
state q 2 . 

After the entire string 1101 has been processed, the machine is in state q 2 , 
which is an accept state. We say that the string 1101 is accepted by the 
machine. 

Consider now the input string 0101010. After having read this string 
from left to right (starting in the start state qi), the machine is in state q 3 . 
Since q 3 is not an accept state, we say that the machine rejects the string 


0101010 . 


We hope you are able to see that this machine accepts every binary string 
that ends with a 1. In fact, the machine accepts more strings: 

• Every binary string having the property that there are an even number 
of 0s following the rightmost 1, is accepted by this machine. 
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• Every other binary string is rejected by the machine. Observe that each 
such string is either empty, consists of Os only, or has an odd number 
of Os following the rightmost 1. 

We now come to the formal definition of a finite automaton: 

Definition 2.2.1 A finite automaton is a 5-tuple M = ( Q , E, <5, g, F), where 

1. Q is a finite set, whose elements are called states , 

2. E is a finite set, called the alphabet ; the elements of E are called symbols , 

3. 8 : Q xT, > Q is & function, called the transition function , 

4. q is an element of Q] it is called the start state , 

5. F is a subset of Q\ the elements of F are called accept states. 

You can think of the transition function 8 as being the “program” of the 
finite automaton M — ( Q , E, 8, q, F). This function tells us what M can do 
in “one step”: 

• Let r be a state of Q and let a be a symbol of the alphabet E. If 
the finite automaton M is in state r and reads the symbol a, then it 
switches from state r to state 8(r, a). (In fact, 8(r, a) may be equal to 
r.) 

The “computer” that we designed in the toll gate example in Section 2.1 
is a finite automaton. For this example, we have Q = (g 0 , cp , g 2 , 9.3, 94, <y 5 }, 
E = {5,10, 25}, the start state is g 0 , F = {g 5 }, and 8 is given by the following 
table: 



5 

10 

25 

Qo 

9i 

92 

95 

Qi 

92 

93 

95 

<72 

93 

94 

95 

<73 

94 

95 

95 

94 

95 

95 

95 

95 

95 

95 

95 


The example given in the beginning of this section is also a finite automa¬ 
ton. For this example, we have Q = {qi, q- 2 , ^ 3 }, E = ( 0 , 1 }, the start state 
is q u F = {g 2 }, and 8 is given by the following table: 
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0 

1 

Qi 

Qi 

92 

Q2 

© 

92 

Q3 

Q‘2 

92 


Let us denote this finite automaton by M. The language of M, denoted 
by L(M), is the set of all binary strings that are accepted by M. As we have 
seen before, we have 

L(M) = {w : w contains at least one 1 and ends with an even number of Os}. 

We now give a formal definition of the language of a finite automaton: 

Definition 2.2.2 Let M — (Q, E, S, q, F ) be a finite automaton and let w — 
iui w 2 ■ ■ ■ w n be a string over E. Define the sequence r 0 , rq,..., r n of states, in 
the following way: 

• r 0 = q, 

• r*+i = S(ri, Wi+i), for i = 0,1,..., n — 1. 

1. If r n G F, then we say that M accepts w. 

2. If r n qL F, then we say that M rejects w. 

In this definition, w may be the empty string, which we denote by e, and 
whose length is zero; thus in the definition above, n = 0. In this case, the 
sequence r 0 ,ri,... ,r n of states has length one; it consists of just the state 
r 0 = q. The empty string is accepted by M if and only if the start state q 
belongs to F. 

Definition 2.2.3 Let M = ( Q,Ti,8,q,F) be a finite automaton. The lan¬ 
guage L(M) accepted by M is defined to be the set of all strings that are 
accepted by M: 

L(M) — {w : w is a string over E and M accepts w }. 

Definition 2.2.4 A language A is called regular , if there exists a finite au¬ 
tomaton M such that A = L(M). 
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We finish this section by presenting an equivalent way of defining the 
language accepted by a finite automaton. Let M = ( Q , E, <5, q, F) be a finite 
automaton. The transition function 8 : Q x E —>• Q tells us that, when M 
is in state r e Q and reads symbol a G E, it switches from state r to state 
5(r, a). Let E* denote the set of all strings over the alphabet E. (E* includes 
the empty string e.) We extend the function 8 to a function 

8 : Q x E* — > Q, 

that is defined as follows. For any state r G Q and for any string w over the 
alphabet E, 

8(r w) J 7 - it w = e, 

' ’ ' \ S(8(r, v),a) if w — va, where v is a string and a e E. 

What is the meaning of this function 81 Let r be a state of Q and let w be 
a string over the alphabet E. Then 

• 8(r, w) is the state that M reaches, when it starts in state r, reads the 
string w from left to right, and uses 8 to switch from state to state. 

Using this notation, we have 

L(M) = {w : w is a string over E and 8(q, w ) ef}. 

2.2.1 A first example of a finite automaton 

Let 

A = {w : w is a binary string containing an odd number of Is}. 

We claim that this language A is regular. In order to prove this, we have to 
construct a finite automaton M such that A = L(M). 

How to construct Ml Here is a first idea: The finite automaton reads the 
input string w from left to right and keeps track of the number of Is it has 
seen. After having read the entire string w, it checks whether this number 
is odd (in which case w is accepted) or even (in which case w is rejected). 
Using this approach, the finite automaton needs a state for every integer 
i > 0, indicating that the number of Is read so far is equal to i. Hence, 
to design a finite automaton that follows this approach, we need an infinite 
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number of states. But, the definition of finite automaton requires the number 
of states to be finite. 

A better, and correct approach, is to keep track of whether the number 
of Is read so far is even or odd. This leads to the following finite automaton: 

• The set of states is Q = {q e , q Q }. If the finite automaton is in state q e , 
then it has read an even number of Is; if it is in state q Q , then it has 
read an odd number of Is. 

• The alphabet is E = {0,1}. 

• The start state is q e , because at the start, the number of Is read by the 
automaton is equal to 0, and 0 is even. 

• The set F of accept states is F = {q 0 }- 

• The transition function 8 is given by the following table: 


0 

q e qe 
q a q 0 


J 

q a 

q e 


This finite automaton M = (Q,T,, 8, q e , F) can also be described by its state 
diagram , which is given in the figure below. The arrow that comes “out of 
the blue” and enters the state q e , indicates that q e is the start state. The 
state depicted with double circles indicates the accept state. 



We have constructed a finite automaton M that accepts the language A. 
Therefore, A is a regular language. 
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2.2.2 A second example of a finite automaton 

Define the language A as 

A = {w : w is a binary string containing 101 as a substring}. 

Again, we claim that A is a regular language. In other words, we claim that 
there exists a finite automaton M that accepts A, i.e., A = L(M ). 

The finite automaton M will do the following, when reading an input 
string from left to right: 

• It skips over all Os, and stays in the start state. 

• At the first 1, it switches to the state “maybe the next two symbols are 
01 ”. 


- If the next symbol is 1, then it stays in the state “maybe the next 
two symbols are 01”. 

— On the other hand, if the next symbol is 0, then it switches to the 
state “maybe the next symbol is 1”. 

* If the next symbol is indeed 1, then it switches to the accept 
state (but keeps on reading until the end of the string). 

* On the other hand, if the next symbol is 0, then it switches 
to the start state, and skips Os until it reads 1 again. 

By defining the following four states, this process will become clear: 

• qi. M is in this state if the last symbol read was 1, but the substring 
101 has not been read. 

• gi 0 : M is in this state if the last two symbols read were 10, but the 
substring 101 has not been read. 

• 5iqi: M is in this state if the substring 101 has been read in the input 
string. 

• q: In all other cases, M is in this state. 

Here is the formal description of the finite automaton that accepts the 
language A: 

• Q = {q,qi, ?io,?ioi}, 
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• S = {0,1}, 

• the start state is q, 

• the set F of accept states is equal to F = {gioi}, and 

• the transition function 8 is given by the following table: 



0 

1 

ff 

<1 

fi 

Qi 

fro 

fi 

fio 

f 

fioi 

fioi 

fioi 

fioi 


The figure below gives the state diagram of the finite automaton M = 
(' Q,Z,8,q,F ). 



This finite automaton accepts the language A consisting of all binary 
strings that contain the substring 101. As an exercise, how would you obtain 
a finite automaton that accepts the complement of A , i.e., the language 
consisting of all binary strings that do not contain the substring 101? 

2.2.3 A third example of a finite automaton 

The finite automata we have seen so far have exactly one accept state. In 
this section, we will see an example of a finite automaton having more accept 
states. 
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Let A be the language 

A = {w e {0,1}* : w has a 1 in the third position from the right}. 

We claim that A is a regular language. To prove this, we have to construct 
a finite automaton M such that A = L(M). At first sight, it seems difficult 
(or even impossible?) to construct such a finite automaton: How does the 
automaton “know” that it has reached the third symbol from the right? It 
is, however, possible to construct such an automaton. The main idea is 
to remember the last three symbols that have been read. Thus, the finite 
automaton has eight states qijk, where i, j, and k range over the two elements 
of {0,1}. If the automaton is in state q^k, then the following hold: 

• If the DFA has read at least three symbols, then the three most recently 
read symbols are ijk. 

• If the DFA has read only two symbols, then these two symbols are jk ; 
moreover, i = 0. 

• If the DFA has read only one symbol, then this symbol is k ; moreover, 

i = j = 0. 

• If the DFA has not read any symbol, then i — j = k = 0. 

The start state is q 0 oo and the set of accept states is {gioo, ?no, 5ioi, 5m}- 
The transition function of the DFA is given by the following state diagram. 
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2.3 Regular operations 

In this section, we define three operations on languages. Later, we will answer 
the question whether the set of all regular languages is closed under these 
operations. Let A and B be two languages over the same alphabet. 

1. The union of A and B is defined as 

AU B = {w : w e A or w E B}. 

2. The concatenation of A and B is defined as 

AB = {ww' : w € A and w' e B}. 

In words, AB is the set of all strings obtained by taking an arbitrary 
string w in A and an arbitrary string w' in B, and gluing them together 
(such that w is to the left of w'). 

3. The star of A is defined as 

A* = {uiu 2 ... u k : k > 0 and & A for all i = 1,2,, k}. 

In words, A* is obtained by taking any finite number of strings in A, and 
gluing them together. Observe that k — 0 is allowed; this corresponds 
to the empty string e. Thus, e E A*. 

To give an example, let A = {0, 01} and B = {1,10}. Then 

AUB = {0,01,1,10}, 

AB = {01,010,011,0110}, 

and 

A* = {e, 0, 01, 00, 001, 010, 0101, 000, 0001,00101,...}. 

As another example, if E = {0,1}, then S* is the set of all binary strings 
(including the empty string). Observe that a string always has a finite length. 

Before we proceed, we give an alternative (and equivalent) definition of 
the star of the language A: Define 
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and, for k > 1, 

A k = AA^ 1 , 

i.e., A k is the concatenation of the two languages A and A k ~ 1 . Then we have 
A* = [J A k . 

k =0 


Theorem 2.3.1 The set of regular languages is closed under the union op¬ 
eration, i.e., if A and B are regular languages over the same alphabet £, then 
A\J B is also a regular language. 

Proof. Since A and B are regular languages, there are finite automata 
Mi = (Qi, E, <$!, qi, Fi) and M 2 = (Q 2 , £, $ 2 , <? 2 , F 2 ) that accept A and B , 
respectively. In order to prove that A U B is regular, we have to construct a 
finite automaton M that accepts A U B. In other words, M must have the 
property that for every string i«eE*, 

M accepts w o M\ accepts w or M 2 accepts w. 

As a first idea, we may think that M could do the following: 

• Starting in the start state q\ of M\ , M “runs” M\ on w. 

• If, after having read w, Mi is in a state of F\, then w <G A, thus 
w G A U B and, therefore, M accepts w. 

• On the other hand, if, after having read w, M\ is in a state that is not 
in F\ , then w ^ A and M “runs” M 2 on w, starting in the start state 
q 2 of M 2 . If, after having read w, M 2 is in a state of F 2 , then we know 
that w G B, thus w & A\J B and, therefore, M accepts w. Otherwise, 
we know that w ^ A U B, and M rejects w. 

This idea does not work, because the finite automaton M can read the input 
string w only once. The correct approach is to run Mi and M 2 simulta¬ 
neously. We define the set Q of states of M to be the Cartesian product 
Qi x Q 2 . If M is in state (ri,r 2 ), this means that 

• if Mi would have read the input string up to this point, then it would 
be in state ri , and 
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• if M 2 would have read the input string up to this point, then it would 
be in state r 2 . 

This leads to the finite automaton M — (Q, E, 8, q, F), where 

• Q = Q\ x Q 2 = {(ri,r 2 ) : r\ E Qi and r 2 E Q 2 }. Observe that 
IQ I = |Qi| x IQ 2 I, which is finite. 

• E is the alphabet of A and B (recall that we assume that A and B are 
languages over the same alphabet). 

• The start state q of M is equal to q = (qi, q 2 ). 

• The set F of accept states of M is given by 

F = {(ri, r 2 ) : r*i € J*i or r 2 E F 2 } = (F 1 x Q 2 ) U (Q 1 x F 2 ). 

• The transition function <5 : Q x E —> Q is given by 

5((ri,r 2 ),a) = (£i(n, a), S 2 (r 2 , a)), 


for all ri E Q 1 , r 2 E Q 2 , and a E E. 

To finish the proof, we have to show that this finite automaton M indeed 
accepts the language AUB. Intuitively, this should be clear from the discus¬ 
sion above. The easiest way to give a formal proof is by using the extended 
transition functions <5i and 6 2 . (The extended transition function has been 
defined after Definition 2.2.4.) Here we go: Recall that we have to prove that 

M accepts w ^ Mi accepts w or M 2 accepts w, 


i.e, 

M accepts w 4^ 6i(qi,w) E F\ or 5 2 (q 2 ,w) E F 2 . 

In terms of the extended transition function 6 of the transition function 6 of 
M, this becomes 

S((qi,q 2 ),w) E F 44 8i(qi,w) E Fi or S 2 (q 2 ,w) E F 2 . (2.1) 

By applying the definition of the extended transition function, as given after 
Definition 2.2.4, to 8 , it can be seen that 

8((qi,q2),w) = (<5i(gi,tu),<$ 2 (g 2 ,«7)). 
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The latter equality implies that (2.1) is true and, therefore, M indeed accepts 
the language AU B. I 

What about the closure of the regular languages under the concatenation 
and star operations? It turns out that the regular languages are closed under 
these operations. But how do we prove this? 

Let A and B be two regular languages, and let and M 2 be finite 
automata that accept A and B, respectively. How do we construct a finite 
automaton M that accepts the concatenation AB1 Given an input string 
u, M has to decide whether or not u can be broken into two strings w and 
w' (i.e., write u as u = ww'), such that w e A and w' e B. In words, M 
has to decide whether or not u can be broken into two substrings, such that 
the first substring is accepted by Mi and the second substring is accepted by 
M 2 . The difficulty is caused by the fact that M has to make this decision by 
scanning the string u only once. If u <G AB, then M has to decide, during 
this single scan , where to break u into two substrings. Similarly, if u ^ AB , 
then M has to decide, during this single scan , that u cannot be broken into 
two substrings such that the first substring is in A and the second substring 
is in B. 

It seems to be even more difficult to prove that A* is a regular language, 
if A itself is regular. In order to prove this, we need a finite automaton that, 
when given an arbitrary input string u , decides whether or not u can be 
broken into substrings such that each substring is in A. The problem is that, 
if u £ A*, the finite automaton has to determine into how many substrings, 
and where, the string u has to be broken; it has to do this during one single 
scan of the string u. 

As we mentioned already, if A and B are regular languages, then both 
AB and A* are also regular. In order to prove these claims, we will introduce 
a more general type of finite automaton. 

The finite automata that we have seen so far are deterministic. This 
means the following: 

• If the finite automaton M is in state r and if it reads the symbol a, 
then M switches from state r to the uniquely defined state 5 (r, a). 

From now on, we will call such a finite automaton a deterministic finite 
automaton (DFA). In the next section, we will define the notion of a nonde- 
terministic finite automaton (NFA). For such an automaton, there are zero 
or more possible states to switch to. At first sight, nondeterministic finite 
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automata seem to be more powerful than their deterministic counterparts. 
We will prove, however, that DFAs have the same power as NFAs. As we will 
see, using this fact, it will be easy to prove that the class of regular languages 
is closed under the concatenation and star operations. 


2.4 Nondeterministic finite automata 

We start by giving three examples of nondeterministic finite automata. These 
examples will show the difference between this type of automata and the 
deterministic versions that we have considered in the previous sections. After 
these examples, we will give a formal definition of a nondeterministic finite 
automaton. 

2.4.1 A first example 

Consider the following state diagram: 


0,1 



You will notice three differences with the finite automata that we have 
seen until now. First, if the automaton is in state q\ and reads the symbol 1, 
then it has two options: Either it stays in state qi, or it switches to state q 2 - 
Second, if the automaton is in state q 2 , then it can switch to state q 3 without 
reading a symbol-, this is indicated by the edge having the empty string e as 
label. Third, if the automaton is in state q 3 and reads the symbol 0, then it 
cannot continue. 

Let us see what this automaton can do when it gets the string 010110 as 
input. Initially, the automaton is in the start state q\. 

• Since the first symbol in the input string is 0, the automaton stays in 
state qi after having read this symbol. 

• The second symbol is 1, and the automaton can either stay in state qi 
or switch to state q 2 . 
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- If the automaton stays in state q x , then it is still in this state after 
having read the third symbol. 

— If the automaton switches to state q 2 , then it again has two op¬ 
tions: 

* Either read the third symbol in the input string, which is 0, 
and switch to state q 3 , 

* or switch to state q 3 , without reading the third symbol. 

If we continue in this way, then we see that, for the input string 010110, 
there are seven possible computations. All these computations are given in 
the figure below. 



Consider the lowest path in the figure above: 

• When reading the first symbol, the automaton stays in state q x . 

• When reading the second symbol, the automaton switches to state q 2 . 

• The automaton does not read the third symbol (equivalently, it “reads” 
the empty string e), and switches to state q 3 . At this moment, the 


Source: www.csitsansar.com 







2.4. Nondeterministic finite automata 


37 


automaton cannot continue: The third symbol is 0, but there is no 
edge leaving q 3 that is labeled 0, and there is no edge leaving q 3 that 
is labeled e. Therefore, the computation hangs at this point. 

From the figure, you can see that, out of the seven possible computations, 
exactly two end in the accept state q 4 (after the entire input string 010110 has 
been read). We say that the automaton accepts the string 010110, because 
there is at least one computation that ends in the accept state. 

Now consider the input string 010. In this case, there are three possible 
computations: 

,010 

1. <?i -> qi -> qi -> q 1 

n 0 1 0 

2. qi —> q 1 —>• q 2 —> q 3 

3. qi A qi —>• q 2 -4 q 3 —> hang 

None of these computations ends in the accept state (after the entire input 
string 010 has been read). Therefore, we say that the automaton rejects the 
input string 010. 

The state diagram given above is an example of a nondeterministic finite 
automaton (NFA). Informally, an NFA accepts a string, if there exists at least 
one path in the state diagram that (i) starts in the start state, (ii) does not 
hang before the entire string has been read, and (iii) ends in an accept state. 
A string for which (i), (ii), and (iii) does not hold is rejected by the NFA. 

The NFA given above accepts all binary strings that contain 101 or 11 as 
a substring. All other binary strings are rejected. 

2.4.2 A second example 

Let A be the language 

A = {w G {0,1}* : w has a 1 in the third position from the right}. 

The following state diagram defines an NFA that accepts all strings that are 
in A, and rejects all strings that are not in A. 


0,1 
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This NFA does the following. If it is in the start state q\ and reads the 
symbol 1 , then it either stays in state q i or it “guesses” that this symbol 
is the third symbol from the right in the input string. In the latter case, 
the NFA switches to state q 2 , and then it “verifies” that there are indeed 
exactly two remaining symbols in the input string. If there are more than 
two remaining symbols, then the NFA hangs (in state 54 ) after having read 
the next two symbols. 

Observe how this guessing mechanism is used: The automaton can only 
read the input string once, from left to right. Hence, it does not know when 
it reaches the third symbol from the right. When the NFA reads a 1, it can 
guess that this is the third symbol from the right; after having made this 
guess, it verifies whether or not the guess was correct. 

In Section 2.2.3, we have seen a DFA for the same language A. Observe 
that the NFA has a much simpler structure than the DFA. 

2.4.3 A third example 

Consider the following state diagram, which defines an NFA whose alphabet 
is { 0 }. 



This NFA accepts the language 

A = {0 fe : k = 0 mod 2 or k = 0 mod 3}, 

where 0 fc is the string consisting of k many Os. (If k = 0, then 0 fc = e.) 
Observe that A is the union of the two languages 

A\ = { 0 fc : k = 0 mod 2 } 


Source: www.csitsansar.com 




2.4. Nondeterministic finite automata 


39 


and 


A 2 = {0 fe : k = 0 mod 3}. 


The NFA basically consists of two DFAs: one of these accepts A x , whereas the 
other accepts A 2 . Given an input string w, the NFA has to decide whether 
or not w G A, which is equivalent to deciding whether or not w G A\ or 
w G A 2 . The NFA makes this decision in the following way: At the start, it 
“guesses” whether (i) it is going to check whether or not w e A\ (i.e., the 
length of w is even), or (ii) it is going to check whether or not w <G A 2 (i.e., 
the length of w is a multiple of 3). After having made the guess, it verifies 
whether or not the guess was correct. If w e A, then there exists a way of 
making the correct guess and verifying that w is indeed an element of A (by 
ending in an accept state). If w A, then no matter which guess is made, 
the NFA will never end in an accept state. 


2.4.4 Definition of nondeterministic finite automaton 

The previous examples give you an idea what nondeterministic finite au¬ 
tomata are and how they work. In this section, we give a formal definition 
of these automata. 

For any alphabet £, we define E e to be the set 
S e = SU {e}. 

Recall the notion of a power set: For any set Q, the power set of Q, denoted 
by V(Q), is the set of all subsets of Q, i.e., 

V{Q) = {R:RCQ}. 

Definition 2.4.1 A nondeterministic finite automaton (NFA) is a 5-tuple 
M = ( Q , E, S, q, F ), where 

1. Q is a finite set, whose elements are called states, 

2. E is a finite set, called the alphabet ; the elements of E are called symbols , 

3. 6 : Q x E e —> V{Q) is a function, called the transition function, 

4. q is an element of Q: it is called the start state, 

5. F is a subset of Q: the elements of F are called accept states. 
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As for DFAs, the transition function <5 can be thought of as the “program” 
of the finite automaton M = (Q,H,8,q,F)-. 

• Let r G Q, and let a G E e . Then 6(r, a ) is a (possibly empty) subset of 
Q. If the NFA M is in state r, and if it reads a (where a may be the 
empty string e), then M can switch from state r to any state in 5(r, a). 
If <5(r, a) = 0, then M cannot continue and the computation hangs. 

The example given in Section 2.4.1 is an NFA, where Q = {qi, q 2 , q 3 , q 4 }, 
E = {0,1}, the start state is q \, the set of accept states is F = {g 4 }, and the 
transition function <5 is given by the following table: 



0 

i 

e 

Qi 


{ 91 , 92 } 

0 

<12 

Us} 

0 

{ 93 } 

Q3 

0 

{ 94 } 

0 

<U 


{ 94 } 

0 


Definition 2.4.2 Let M — (Q,'E,6,q,F) be an NFA, and let w G E*. We 
say that M accepts w, if w can be written as w = yiy 2 ... y m , where yi e E e 
for all i with 1 < % < m, and there exists a sequence r 0 , ri, ..., r m of states 
in Q, such that 

• r 0 = q, 

• r i+ 1 G 5(ri, y i+ 1 ), for i = 0,1 , ..., m — 1 , and 


• r m G F. 

Otherwise, we say that M rejects the string w. 

The NFA in the example in Section 2.4.1 accepts the string 01100. This 
can be seen by taking 

• tv = OlelOO = 2/12/22/32/42/52/6, and 


• r 0 = qi, ri = qi, r 2 = q 2 , r 3 = q 3 , r 4 = ? 4 , r 5 = g 4 , and r 6 = q 4 . 

Definition 2.4.3 Let M = (Q, E, 6 , q, F) be an NFA. The language L(M ) 
accepted by M is defined as 

L(M) = {w G E* : M accepts w }. 
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2.5 Equivalence of DFAs and NFAs 


You may have the impression that nondeterministic finite automata are more 
powerful than deterministic finite automata. In this section, we will show 
that this is not the case. That is, we will prove that a language can be 
accepted by a DFA if and only if it can be accepted by an NFA. In order 
to prove this, we will show how to convert an arbitrary NFA to a DFA that 
accepts the same language. 

What about converting a DFA to an NFA? Well, there is (almost) nothing 
to do, because a DFA is also an NFA. This is not quite true, because 

• the transition function of a DFA maps a state and a symbol to a state, 


whereas 


• the transition function of an NFA maps a state and a symbol to a set 
of zero or more states. 

The formal conversion of a DFA to an NFA is done as follows: Let M = 
(' Q,Ti,5,q,F ) be a DFA. Recall that S is a function S : Q x E —>• Q. We 
define the function S' : Q x E £ —>• V(Q) as follows. For any r e Q and for 
any a e E e , 



Then N = (Q, E, 6', q, F ) is an NFA, whose behavior is exactly the same as 
that of the DFA M; the easiest way to see this is by observing that the state 
diagrams of M and N are equal. Therefore, we have L(M) = L(N). 

In the rest of this section, we will show how to convert an NFA to a DFA: 

Theorem 2.5.1 Let N = (Q , E, 6 , q , F) be a nondeterministic finite automa¬ 
ton. There exists a deterministic finite automaton M, such that L(M) = 
L(N). 

Proof. Recall that the NFA N can (in general) perform more than one 
computation on a given input string. The idea of the proof is to construct a 
DFA M that runs all these different computations simultaneously. (We have 
seen this idea already in the proof of Theorem 2.3.1.) To be more precise, 
the DFA M will have the following property: 

• the state that M is in after having read an initial part of the input 
string corresponds exactly to the set of all states that N can reach 
after having read the same part of the input string. 
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We start by presenting the conversion for the case when N does not 
contain e-transitions. In other words, the state diagram of N does not contain 
any edge that has e as a label. (Later, we will extend the conversion to the 
general case.) Let the DFA M be defined as M = (Q / , E, S', q', F'), where 

• the set Q' of states is equal to Q' = V{Q)\ observe that IQ'I = 2^1, 

• the start state q' is equal to q' = {(/}; so M has the “same” start state 
as A, 

• the set F' of accept states is equal to the set of all elements R of Q' 
having the property that R contains at least one accept state of N , i.e., 

F' = {R G Q : R n F + 0}, 

• the transition function S' : Q' x E —>• Q' is defined as follows: For each 
R G Q' and for each a e E, 

S\R, a) = S(r, a). 

r£R 

Let us see what the transition function S' of M does. First observe that, 
since N is an NFA, <5(r, a) is a subset of Q. This implies that S'{R, a) is the 
union of subsets of Q and, therefore, also a subset of Q. Hence, S'(R, a) is 
an element of Q'. 

The set S (r, a) is equal to the set of all states of the NFA N that can be 
reached from state r by reading the symbol a. We take the union of these 
sets S(r,a), where r ranges over all elements of R, to obtain the new set 
S'(R , a). This new set is the state that the DFA M reaches from state R, by 
reading the symbol a. 

In this way, we obtain the correspondence that was given in the beginning 
of this proof. 

After this warming-up, we can consider the general case. In other words, 
from now on, we allow e-transitions in the NFA N. The DFA M is defined as 
above, except that the start state q' and the transition function S' have to be 
modified. Recall that a computation of the NFA N consists of the following: 

1. Start in the start state q and make zero or more e-transitions. 

2. Read one “real” symbol of E and move to a new state (or stay in the 
current state). 
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3. Make zero or more e-transitions. 

4. Read one “real” symbol of E and move to a new state (or stay in the 
current state). 

5. Make zero or more e-transitions. 

6. Etc. 

The DFA M will simulate this computation in the following way: 

• Simulate 1. in one single step. As we will see below, this simulation is 
implicitly encoded in the definition of the start state q' of M. 

• Simulate 2. and 3. in one single step. 

• Simulate 4. and 5. in one single step. 

• Etc. 

Thus, in one step, the DFA M simulates the reading of one “real” symbol of 
E, followed by making zero or more e-transitions. 

To formalize this, we need the notion of e- closure. For any state r of the 
NFA N, the e-closure of r, denoted by C e (r), is defined to be the set of all 
states of N that can be reached from r, by making zero or more e-transitions. 
For any state R of the DFA M (hence, it C Q), we define 

C,(R) = U 

reR 

How do we define the start state q' of the DFA Ml Before the NFA N 
reads its first “real” symbol of E, it makes zero or more e-transitions. In 
other words, at the moment when N reads the first symbol of E, it can be 
in any state of C e (q). Therefore, we define q' to be 

<J = C e {q) = C e {{q}), 

How do we define the transition function 5' of the DFA Ml Assume that 
M is in state R, and reads the symbol a. At this moment, the NFA N would 
have been in any state r of R. By reading the symbol a, N can switch to 
any state in 5(r, a), and then make zero or more e-transitions. Hence, the 
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NFA can switch to any state in the set C e (8(r, a)). Based on this, we define 
8'(R , a) to be 

8’(R,a)= (J C e (S(r, a)). 

r€R 

To summarize, the NFA N = (Q, E, 5, q, F ) is converted to the DFA 
M = ( Q ', E, 5', q 1 , F'), where 

• Q' = V{Q), 

• q' = Ce{{q}), 

• F' = {Re Q' : RnF^®}, 

• 8' : Q' x E —> Q' is defined as follows: For each R, <G Q' and for each 
a G E, 

5'{R,a) - (J C e (8(r,a)). 

r£R 


The results proved until now can be summarized in the following theorem. 

Theorem 2.5.2 Let A be a language. Then A is regular if and only if there 
exists a nondeterministic finite automaton that accepts A. 

2.5.1 An example 

Consider the NFA N = ( Q , E, <5, q, F), where Q = {1, 2, 3}, E = {a, 6}, q = 1, 
F = {2}, and 8 is given by the following table: 



a 

b 

e 

1 

{3} 

0 

{2} 

2 

{1} 

0 

0 

3 

{2} 

{2,3} 

0 


The state diagram of A is as follows: 
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We will show how to convert this NFA A to a DFA M that accepts the 
same language. Following the proof of Theorem 2.5.1, the DFA M is specified 
by M — (Q\ E, S', q', F'), where each of the components is defined below. 

• Q' = V{Q). Hence, 

Of = W {1}, {2}, {3}, { 1 , 2}, { 1 , 3}, {2, 3}, { 1 , 2, 3}}. 

• q' = (7 e ({g}). Hence, the start state q 1 of M is the set of all states of 
N that can be reached from TV’s start state q = 1, by making zero or 
more e-transitions. We obtain 

q' = C e ({q}) = C e ({ 1}) = {1,2}. 


• F' = {R G Q' : R fl F 0}. Hence, the accept states of M are those 
states that contain the accept state 2 of N. We obtain 

= {{ 2 }, { 1 , 2 }, { 2 , 3 }, { 1 , 2 , 3 }}. 


• S' : Q' x E —>• Q' is defined as follows: For each R G Q' and for each 
a G E, 

S\R,a) = [J C e (S(r, a)). 

reR 
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In this example S' is given by 


<5'(0, a) = 0 
5'({l},a) = {3} 
a ) — (!) 2 } 
5'({3},a) = {2} 
<5'({1, 2}, a) - {1,2,3} 
<5'({l,3},a) = {2,3} 
^({2, 3}, a) = {1,2} 
<5'({1, 2, 3}, a) - {1,2,3} 


5 '( 0 , 6 ) = 0 

5 '({ 1 }, 6 ) = 0 
5'({2},6) = 0 
5'({3},6) = {2,3} 
*'({!, 2}, 6) = 0 
ff({l,3},&) = {2,3} 
<5'({2,3}, 6) = {2,3} 
<5'({1,2,3}, 6) - {2,3} 


The state diagram of the DFA M is as follows: 



We make the following observations: 


Source: www.csitsansar.com 








2.6. Closure under the regular operations 


47 


• The states {1} and {1, 3} do not have incoming edges. Therefore, these 
two states cannot be reached from the start state {1,2}. 

• The state {3} has only one incoming edge; it comes from the state 
{1}. Since {1} cannot be reached from the start state, {3} cannot be 
reached from the start state. 

• The state {2} has only one incoming edge; it comes from the state 
{3}. Since {3} cannot be reached from the start state, {2} cannot be 
reached from the start state. 

Hence, we can remove the four states {1}, {2}, {3}, and {1,3}. The 
resulting DFA accepts the same language as the DFA above. This leads 
to the following state diagram, which depicts a DFA that accepts the same 
language as the NFA N: 



b 



Source: www.csitsansar.com 






48 


Chapter 2. Finite Automata and Regular Languages 


2.6 Closure under the regular operations 

In Section 2.3, we have defined the regular operations union, concatenation, 
and star. We proved in Theorem 2.3.1 that the union of two regular lan¬ 
guages is a regular language. We also explained why it is not clear that the 
concatenation of two regular languages is regular, and that the star of a reg¬ 
ular language is regular. In this section, we will see that the concept of NFA, 
together with Theorem 2.5.2, can be used to give a simple proof of the fact 
that the regular languages are indeed closed under the regular operations. 
We start by giving an alternative proof of Theorem 2.3.1: 

Theorem 2.6.1 The set of regular languages is closed under the union op¬ 
eration, i.e., if A\ and A 2 are regular languages over the same alphabet E, 
then Ax U A 2 is also a regular language. 

Proof. Since Ax is regular, there is, by Theorem 2.5.2, an NFA M x = 
{Qx, E, Si,qi, Fj), such that A l — L{Mx). Similarly, there is an NFA M 2 = 
( Q 2 , E, 5 2 , q 2 , F 2 ), such that A 2 = L{M 2 ). We may assume that Qx D Q 2 = 0, 
because otherwise, we can give new “names” to the states of Qx and Q 2 . 
From these two NFAs, we will construct an NFA M — (Q, E, 5, g 0 , F), such 
that L{M) = Ax U A 2 . The construction is illustrated in Figure 2.1. The 
NFA M is defined as follows: 

1. Q = {go} UQiU Q 2 , where g 0 is a new state. 

2. g 0 is the start state of M. 

3. F = FxUF 2 . 

4. 6 : Q x E e —> V{Q) is defined as follows: For any r <G Q and for any 
a G E e , 



S 2 (r,a ) if reQ 2 , 

{gi, g 2 } if r = g 0 and a = e, 
0 if r = g 0 and a ^ e. 
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Figure 2.1: The NFA M accepts L(Mf) U L(M 2 ). 


Theorem 2.6.2 The set of regular languages is closed under the concatena¬ 
tion operation, i.e., if A 1 and A 2 are regular languages over the same alphabet 
£, then A\A 2 is also a regular language. 

Proof. Let = (Qi, £, g x , Ff) be an NFA, such that A 1 = L(M X ). 
Similarly, let M 2 = {Q 2 , E, S 2 , q 2 , F 2 ) be an NFA, such that A 2 = L(M 2 ). 
As in the proof of Theorem 2.6.1, we may assume that Qi C\ Q 2 = 0. We 
will construct an NFA M = (Q,T,,S,q 0 , F), such that L(M ) = A 1 A 2 . The 
construction is illustrated in Figure 2.2. The NFA M is defined as follows: 

1. Q = Q!UQ 2 . 

2. q 0 = q 1 . 

3. F = F 2 . 
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4. S : Q x E e —>• V(Q) is defined as follows: For any r e Q and for any 
a € E e , 

{ 5i(r, a) if r G Qi and r # Fi, 

8\ (r, a) if r e Fi and a ^ e, 

5i(r, a) U {g 2 } if r e .Fi and a = e, 

S 2 (r,a ) if r G (52- 


Theorem 2.6.3 The set of regular languages is closed under the star oper¬ 
ation, i.e., if A is a regular language, then A* is also a regular language. 

Proof. Let E be the alphabet of A and let N — (Qi, E, <5i, q\, Ff) be an 
NFA, such that A = L(N). We will construct an NFA M — (Q, E, 6 , g 0 , F), 
such that L(M ) = A*. The construction is illustrated in Figure 2.3. The 
NFA M is defined as follows: 
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Figure 2.3: The NFA M accepts (L(N))*. 


1. Q = {g 0 } U Q i, where g 0 is a new state. 

2. g 0 is the start state of M. 


3. F = {g 0 } U Fi. (Since eel*, g 0 has to be an accept state.) 


4. 5 : Q x £ e —y V{Q) is defined as follows: For any r <G Q and for any 
a G E e , 


a) 

Si(r, a) 

8i(r, a) U {gi} 

tii} 

0 


if r e Qi and r 0 F\. 
if r e Fi and a ^ e, 
if r G Fi and a = e, 
if r = g 0 and a = e, 
if r = g 0 and a ^ e. 


In the final theorem of this section, we mention (without proof) two more 
closure properties of the regular languages: 

Theorem 2.6.4 The set of regular languages is closed under the complement 
and intersection operations: 

1. If A is a regular language over the alphabet H, then the complement 
A = {w G £* : w A} 
is also a regular language. 
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2. If A 1 and A 2 are regular languages over the same alphabet E, then the 
intersection 


A 1 fl A 2 = {w e E* : w G A\ and w e A 2 } 
is also a regular language. 


2.7 Regular expressions 


In this section, we present regular expressions, which are a means to describe 
languages. As we will see, the class of languages that can be described by 
regular expressions coincides with the class of regular languages. 

Before formally defining the notion of a regular expression, we give some 
examples. Consider the expression 

(oui)oi*. 

The language described by this expression is the set of all binary strings 

1. that start with either 0 or 1 (this is indicated by (0 U 1)), 

2. for which the second symbol is 0 (this is indicated by 0), and 

3. that end with zero or more Is (this is indicated by 1*). 

That is, the language described by this expression is 


{ 00 , 001 , 0011 , 00111 ,..., 10 , 101 , 1011 , 10111 ,...}. 


Here are some more examples (in all cases, the alphabet is {0,1}): 


• The language { w : w contains exactly two 0s} is described by the ex¬ 
pression 


1 * 01 * 01 *. 


• The language {w 
pression 


: w contains at least two 0s} is described by the ex- 
(0U1)*0(0U1)*0(0U1)*. 


• The language {w 
pression 


1011 is a substring of w} is described by the ex- 
(0U1)*1011(0U1)*. 
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• The language {w : the length of w is even} is described by the expres¬ 
sion 

((0 U 1)(0 U 1))*. 

• The language {w : the length of w is odd} is described by the expres¬ 
sion 

(0U1)((0U1)(0U1))*. 

• The language {1011, 0} is described by the expression 

1011 U0. 

• The language {w : the first and last symbols of w are equal} is de¬ 
scribed by the expression 

0(0U1)*0U1(0U1)*1U0U1. 

After these examples, we give a formal (and inductive) definition of regular 
expressions: 

Definition 2.7.1 Let E be a non-empty alphabet. 

1. e is a regular expression. 

2. 0 is a regular expression. 

3. For each a e E, a is a regular expression. 

4. If Ri and R 2 are regular expressions, then R\ U R 2 is a regular expres¬ 
sion. 

5. If Ri and R 2 are regular expressions, then R\R 2 is a regular expression. 

6. If R is a regular expression, then R* is a regular expression. 

You can regard 1., 2., and 3. as being the “building blocks” of regular 
expressions. Items 4., 5., and 6. give rules that can be used to combine 
regular expressions into new (and “larger”) regular expressions. To give an 
example, we claim that 

(0U1)*101(0U1)* 

is a regular expression (where the alphabet E is equal to {0,1}). In order 
to prove this, we have to show that this expression can be “built” using the 
“rules” given in Definition 2.7.1. Here we go: 
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• By 3., 0 is a regular expression. 

• By 3., 1 is a regular expression. 

• Since 0 and 1 are regular expressions, by 4., OUl is a regular expression. 

• Since OUl is a regular expression, by 6., (OUl)* is a regular expression. 

• Since 1 and 0 are regular expressions, by 5., 10 is a regular expression. 

• Since 10 and 1 are regular expressions, by 5., 101 is a regular expression. 

• Since (0 U 1)* and 101 are regular expressions, by 5., (0 U 1)*101 is a 
regular expression. 

• Since (0 U 1)*101 and (0 U 1)* are regular expressions, by 5., (0 U 
l)*101(0Ul)* is a regular expression. 

Next we define the language that is described by a regular expression: 

Definition 2.7.2 Let E be a non-empty alphabet. 

1. The regular expression e describes the language {e}. 

2. The regular expression 0 describes the language 0. 

3. For each a e E, the regular expression a describes the language {a}. 

4. Let R\ and R 2 be regular expressions and let Li and L 2 be the lan¬ 
guages described by them, respectively. The regular expression R] U R 2 
describes the language Li U L 2 . 

5. Let R\ and R 2 be regular expressions and let Li and L 2 be the languages 
described by them, respectively. The regular expression R\R 2 describes 
the language LiL 2 . 

6. Let R be a regular expression and let L be the language described by 
it. The regular expression R* describes the language L*. 

We consider some examples: 

• The regular expression (0 U e) (1 U e) describes the language {01, 0,1, e}. 
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• The regular expression 0 U e describes the language {0, e}, whereas the 
regular expression 1* describes the language {e, 1,11, 111,..There¬ 
fore,the regular expression (0 U e)l* describes the language 

{0,01, Oil, 0111,..., e,l,ll,lll,...}. 

Observe that this language is also described by the regular expression 
01* U 1*. 

• The regular expression 1*0 describes the empty language, i.e., the lan¬ 
guage 0. (You should convince yourself that this is correct.) 

• The regular expression 0* describes the language {e}. 

Definition 2.7.3 Let R\ and R 2 be regular expressions and let L\ and L 2 
be the languages described by them, respectively. If L\ = L 2 (i.e., R,\ and 
R 2 describe the same language), then we will write Ri — R 2 . 

Hence, even though (0Ue)l* and 01* Ul* are different regular expressions, 
we write 

(0Ue)l* = 01* Ul*, 

because they describe the same language. 

In Section 2.8.2, we will show that every regular language can be described 
by a regular expression. The proof of this fact is purely algebraic and uses 
the following algebraic identities involving regular expressions. 

Theorem 2.7.4 Let R\, R 2 , and R 3 be regular expressions. The following 
identities hold: 

1 . Ri0 = 0Ri = 0. 


1 R\£ — eR\ — R\. 


3. R 1 U0 = 0URi = R 1 . 

4- R\ U R\ = R\. 

5. R!UR 2 = R 2 U Ri. 

6. R\{R 2 U Rf) — R\R 2 U R\R%. 
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7. U R 2 )R3 — R\R% U R 2 R^. 

8. Rl(R 2 R 3 ) = (RlR 2 )R3- 

9. 0* =: e. 

10. e* = 6. 

11. (eURi)* = RJ. 

72. (e U Ri)(e U Ri)* = R*. 

13. R\{e U Ri) = (e U Ri)R* = R*. 

If. r\r 2 U R 2 — r\r 2 . 

15. Ri(R 2 Ri)* = (RiR 2 )*Ri. 

16. (Ri U R 2 )* = (R\R 2 )*R\ = (R^Ri)*R^. 

We will not present the (boring) proofs of these identities, but urge you 
to convince yourself informally that they make perfect sense. To give an 
example, we mentioned above that 

(OUe)l* = 01* U 1*. 

We can verify this identity in the following way: 

(0 U e)l* = 01* U el* (by identity 7) 

= 01* U 1* (by identity 2) 

2.8 Equivalence of regular expressions and reg¬ 
ular languages 

In the beginning of Section 2.7, we mentioned the following result: 

Theorem 2.8.1 Let L be a language. Then L is regular if and only if there 
exists a regular expression that describes L. 

The proof of this theorem consists of two parts: 
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• In Section 2.8.1, we will prove that every regular expression describes 
a regular language. 

• In Section 2.8.2, we will prove that every DFA M can be converted to 
a regular expression that describes the language L(M). 

These two results will prove Theorem 2.8.1. 

2.8.1 Every regular expression describes a regular lan¬ 


guage 


Let R be an arbitrary regular expression over the alphabet E. We will prove 
that the language described by R is a regular language. The proof is by 
induction on the structure of R (i.e., by induction on the way R is “built” 
using the “rules” given in Definition 2.7.1). 

The first base case: Assume that R = e. Then R describes the lan¬ 
guage {e}. In order to prove that this language is regular, it suffices, by 
Theorem 2.5.2, to construct an NFA M = (Q , E, 8, q, F ) that accepts this 
language. This NFA is obtained by defining Q = {g}, q is the start state, 
F = {g}, and 8(q,a) — 0 for all a G E e . The figure below gives the state 
diagram of M: 



The second base case: Assume that R = 0. Then R describes the language 
0. In order to prove that this language is regular, it suffices, by Theorem 2.5.2, 
to construct an NFA M = (Q, E, 8, q, F) that accepts this language. This 
NFA is obtained by defining Q = {g}, g is the start state, F = 0, and 
8(q, a) — 0 for all a G E e . The figure below gives the state diagram of M: 



The third base case: Let a G E and assume that R = a. Then R describes 
the language {a}. In order to prove that this language is regular, it suffices, 
by Theorem 2.5.2, to construct an NFA M = (Q, E, 8, q \, F) that accepts 
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this language. This NFA is obtained by defining Q = {q x , q 2 }, q\ is the start 
state, F = {g 2 }, and 


S(q h a) = {q 2 }, 

S(q lt b) = 0 for all 6eS f \ {a}, 
S(q 2 , b) = 0 for all b e £ e . 

The figure below gives the state diagram of M: 



The first case of the induction step: Assume that R = R\ U R 2 , where 
Ri and R 2 are regular expressions. Let L 1 and L 2 be the languages described 
by R\ and R 2 , respectively, and assume that L\ and L 2 are regular. Then R 
describes the language Li U L 2 , which, by Theorem 2.6.1, is regular. 

The second case of the induction step: Assume that R = R±R 2 , where 
Ri and R 2 are regular expressions. Let Li and L 2 be the languages described 
by R\ and R 2 , respectively, and assume that Li and L 2 are regular. Then R 
describes the language L\L 2 , which, by Theorem 2.6.2, is regular. 

The third case of the induction step: Assume that R = (Ri)*, where 
Ri is a regular expression. Let Li be the language described by R,\ and 
assume that Li is regular. Then R describes the language (Li)*, which, by 
Theorem 2.6.3, is regular. 

This concludes the proof of the claim that every regular expression de¬ 
scribes a regular language. 

To give an example, consider the regular expression 

(ab U a)*, 

where the alphabet is {a, b}. We will prove that this regular expression de¬ 
scribes a regular language, by constructing an NFA that accepts the language 
described by this regular expression. Observe how the regular expression is 
“built”: 

• Take the regular expressions a and b, and combine them into the regular 
expression ab. 
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• Take the regular expressions ab and a , and combine them into the 
regular expression ab U a. 

• Take the regular expression aiUs, and transform it into the regular 
expression (a&Ua)*. 

First, we construct an NFA Mi that accepts the language described by 
the regular expression a: 

Mi 

Next, we construct an NFA M 2 that accepts the language described by 
the regular expression b: 

m 2 




Next, we apply the construction given in the proof of Theorem 2.6.2 to 
Mi and M 2 . This gives an NFA M 3 that accepts the language described by 
the regular expression ab: 


Mi 



Next, we apply the construction given in the proof of Theorem 2.6.1 to 
M 3 and Mi . This gives an NFA M 4 that accepts the language described by 
the regular expression ab U a: 



Finally, we apply the construction given in the proof of Theorem 2.6.3 
to M 4 . This gives an NFA M 5 that accepts the language described by the 
regular expression (ab U a)*: 
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2.8.2 Converting a DFA to a regular expression 

In this section, we will prove that every DFA M can be converted to a regular 
expression that describes the language L(M). In order to prove this result, 
we need to solve recurrence relations involving languages. 

Solving recurrence relations 

Let E be an alphabet, let B and C be “known” languages in E* such that 
e 0 B, and let L be an “unknown” language such that 

L = BL U C. 

Can we “solve” this equation for LI That is, can we express L in terms of 
B and Cl 

Consider an arbitrary string u in L. We are going to determine how u 
looks like. Since u G L and L = BL U (7, we know that u is a string in 
BLUC. Hence, there are two possibilities for u. 

1. u is an element of C. 

2. u is an element of BL. In this case, there are strings b 6 B and v G L 
such that u = bv. Since e ^ B, we have b ^ e and, therefore, |u| < |m|. 
(Recall that |u| denotes the length, i.e., the number of symbols, of the 
string v.) Since v is a string in L, which is equal to BL U (7, v is a 
string in BL U (7. Hence, there are two possibilities for v. 
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(a) v is an element of C. In this case, 

u = bv, where b G B and v G C\ thus, u G BC. 

(b) v is an element of BL. In this case, there are strings b' G B and 

w G L such that v ■ b'w. Since e ^ B, we have b' ^ e and, 

therefore, |iu| < |n|. Since re is a string in L, which is equal to 
BLUC , w is a string in BLUC. Hence, there are two possibilities 
for w. 

i. w is an element of C. In this case, 

u = bb'w , where b,b' G B and wGC; thus, u G BBC. 

ii. w is an element of BL. In this case, there are strings b" G B 

and x G L such that w = b"x. Since e ^ B, we have b" ^ e 

and, therefore, |x| < | w |. Since x is a string in L, which is 
equal to BL U (7, x is a string in BL Li C. Hence, there are 
two possibilities for x. 

A. x is an element of C. In this case, 

u = bb'V'x , where b, b ', b" G B and x G C\ thus, u G BBBC. 

B. x is an element of BL. Etc., etc. 

This process hopefully convinces you that any string u in L can be written 
as the concatenation of zero or more strings in B, followed by one string in 

C. In fact, L consists of exactly those strings having this property: 

Lemma 2.8.2 Let E be an alphabet, and let B, C, and L be languages in 
E* such that e 0 B and 

L = BLUC. 


Proof. First, we show that B*C C L. Let u be an arbitrary string in B*C. 
Then u is the concatenation of k strings of B, for some k > 0, followed by 
one string of C. We proceed by induction on k. 

The base case is when k = 0. In this case, u is a string in C. Hence, u is 
a string in BL U C. Since BL U C = L , it follows that u is a string in L. 
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Now let k > 1. Then we can write u = vwc, where v is a string in B, 
w is the concatenation of k — 1 strings of B, and c is a string of C. Define 
y — wc. Observe that y is the concatenation of k — 1 strings of B followed 
by one string of C. Therefore, by induction, the string y is an element of L. 
Hence, u = vy, where v is a string in B and y is a string in L. This shows 
that u is a string in BL. Hence, u is a string in BL U C. Since BL U C = L, 
it follows that u is a string in L. This completes the proof that B*C C L. 

It remains to show that L C B*C. Let u be an arbitrary string in L, 
and let £ be its length (i.e., £ is the number of symbols in u). We prove by 
induction on £ that u is a string in B*C. 

The base case is when £ — 0. Then u — e. Since and L = BL U C, 

u is a string in BL U C. Since e ^ B, u cannot be a string in BL. Hence, u 
must be a string in C. Since C C B*C, it follows that u is a string in B*C. 

Let £ > 1. If u is a string in C , then u is a string in B*C and we are done. 
So assume that u is not a string in C. Since u e L and L = BL U C', u is a 
string in BL. Hence, there are strings b e B and v e L such that u = bv. 
Since e ^ B, the length of b is at least one; hence, the length of v is less than 
the length of u. By induction, v is a string in B*C. Hence, u = bv, where 
b G B and v E B*C. This shows that u E B(B*C). Since B(B*C ) C B*C, 
it follows that u E B*C. ■ 


The conversion 

We will now use Lemma 2.8.2 to prove that every DFA can be converted to 
a regular expression. 

Let M — (Q, S, 5, q , F) be an arbitrary deterministic finite automaton. 
We will show that there exists a regular expression that describes the lan¬ 
guage L(M). 

For each state r E Q, we define 

L r = {w E £* : the path in the state diagram of M that starts 
in state r and that corresponds to w ends in a 
state of F }. 

In words, L r is the language accepted by M, if r were the start state. 

We will show that each such language L r can be described by a regular 
expression. Since L(M) = L q , this will prove that L(M) can be described by 
a regular expression. 


Source: www.csitsansar.com 



2.8. Equivalence of regular expressions and regular languages 63 


The basic idea is to set up equations for the languages L r , which we then 
solve using Lemma 2.8.2. We claim that 

L r = |J a • L 5{ria) if r<£F. (2.2) 

a£S 

Why is this true? Let w be a string in L r . Then the path P in the state 
diagram of M that starts in state r and that corresponds to w ends in a 
state of F. Since r F, this path contains at least one edge. Let r' be the 
state that follows the first state (i.e., r ) of P. Then r' = S(r,b) for some 
symbol b £ £. Hence, b is the first symbol of w. Write w = bv, where v is 
the remaining part of w. Then the path P' = P\{r} in the state diagram 
of M that starts in state r' and that corresponds to v ends in a state of F. 
Therefore, v £ L r > = Ls( r ,b) ■ Hence, 

web - L S (r t b) Q U a ' L S(r,a) ■ 

Conversely, let w be a string in (J aeE a ■ L^ ra) . Then there is a symbol b <G E 
and a string v 6 L s ^ such that w — bv. Let P' be the path in the state 
diagram of M that starts in state S(r,b) and that corresponds to v. Since 
v G Ls(r,b)i this path ends in a state of F. Let P be the path in the state 
diagram of M that starts in r, follows the edge to S(r, b), and then follows P'. 
This path P corresponds to w and ends in a state of F. Therefore, w e L r . 
This proves the correctness of (2.2). 

Similarly, we can prove that 

L r = e U I [J a ■ L S (r,a) J if r £ F. (2.3) 

Vaes J 

So we now have a set of equations in the “unknowns” L r , for r £ Q. The 
number of equations is equal to the size of Q. In other words, the number 
of equations is equal to the number of unknowns. The regular expression for 
L(M) = L q is obtained by solving these equations using Lemma 2.8.2. 

Of course, we have to convince ourselves that these equations have a so¬ 
lution for any given DFA. Before we deal with this issue, we give an example. 

An example 

Consider the deterministic finite automaton M — (Q, E, 5, g 0 , F), where Q = 
{g 0 , gi, g 2 }, £ — {a, b}, g 0 is the start state, F = {g 2 }, and S is given in the 
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state diagram below. We show how to obtain the regular expression that 
describes the language accepted by M. 



For this case, (2.2) and (2.3) give the following equations: 



In the third equation, L q2 is expressed in terms of L qo and L qi . Hence, if we 
substitute the third equation into the first one, and use Theorem 2.7.4, then 
we get 

L qo = a ■ L qo U b ■ (e U a ■ L qi U b ■ L qo ) 

= (a U bb ) • L qo U ba ■ L qi U b. 

We obtain the following set of equations. 

f L qo = (a U bb) ■ L qo U ba ■ L qi U b 
\ L q i = b ■ L qi U a ■ L qo 

Let L = L qi , B = b, and C — a ■ L qo . Then e ^ B and the second equation 
reads L = BL U C. Hence, by Lemma 2.8.2, 

L qi = L = B*C - b*a ■ L qo . 
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If we substitute L qi into the first equation, then we get (again using Theo¬ 
rem 2.7.4) 


L qo = (a U bb) ■ L qo U ba ■ b*a ■ L qo U b 
= (a U bb U bab*a)L qo U b. 


Again applying Lemma 2.8.2, this time with L = L qo , B = a U bb U half a and 
(7 = 6, gives 


L qo = (aUbbU bob*a)* b. 


Thus, the regular expression that describes the language accepted by M is 


(a U bb U bob*a)* b. 


Completing the correctness of the conversion 

It remains to prove that, for any DFA, the system of equations (2.2) and (2.3) 
can be solved. This will follow from the following (more general) lemma. 
(You should verify that the equations (2.2) and (2.3) are in the form as 
specified in this lemma.) 

Lemma 2.8.3 Let n > 1 be an integer and, for 1 < i < n and 1 < j < n, 
let Bij and Ci be regular expressions such that e Bij. Let L 1 , L 2 ,..., L n be 
languages that satisfy 



Then Li can be expressed as a regular expression only involving the regular 
expressions B tj and Ci. 

Proof. The proof is by induction on n. The base case is when n — 1. In 
this case, we have 


L\ — B\\L\ U C\. 


Since e 0 Bn, it follows from Lemma 2.8.2 that Li = B* vx C\. This proves 
the base case. 
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Let n > 2 and assume the lemma is true for n — 1. We have 


Ln = (U B "AjUC„ 

= .. (ij ) •: 


Since e ^ B nn , it follows from Lemma 2.8.2 that 

Ln = B* nn (J\jB nj L^juC n 

= B* nn ^(J B nj L^j U B* nn C n 
= ( y B* nn B nj LA U B* n C n 


By substituting this equation for L n into the equations for Lj, 1 < i < n — 1, 
we obtain 


L, = 

= u (jj BijLj ) U Ci 


U {B in B* nn B nj U B tj ) Lj U B in B* nn C n U C % . 


Thus, we have obtained n — 1 equations in L%, L 2 ,.. ■, T n _i- Since e 0 
Bi n B^ n B n j U B tJ , it follows from the induction hypothesis that L\ can be 
expressed as a regular expression only involving the regular expressions 
and Ci . ■ 
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2.9 The pumping lemma and nonregular lan¬ 
guages 

In the previous sections, we have seen that the class of regular languages is 
closed under various operations, and that these languages can be described by 
(deterministic or nondeterministic) finite automata and regular expressions. 
These properties helped in developing techniques for showing that a language 
is regular. In this section, we will present a tool that can be used to prove 
that certain languages are not regular. Observe that for a regular language, 

1. the amount of memory that is needed to determine whether or not a 
given string is in the language is finite and independent of the length 
of the string, and 

2. if the language consists of an infinite number of strings, then this lan¬ 
guage should contain infinite subsets having a fairly repetitive struc¬ 
ture. 

Intuitively, languages that do not follow 1. or 2. should be nonregular. For 
example, consider the language 

{0 n r : n > 0}. 

This language should be nonregular, because it seems unlikely that a DFA can 
remember how many Os it has seen when it has reached the border between 
the Os and the Is. Similarly the language 

{0 n : n is a prime number} 

should be nonregular, because the prime numbers do not seem to have any 
repetitive structure that can be used by a DFA. To be more rigorous about 
this, we will establish a property that all regular languages must possess. 
This property is called the pumping lemma. If a language does not have this 
property, then it must be nonregular. 

The pumping lemma states that any sufficiently long string in a regular 
language can be pumped , i.e., there is a section in that string that can be 
repeated any number of times, so that the resulting strings are all in the 
language. 
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Theorem 2.9.1 (Pumping Lemma for Regular Languages) Let A be 

a regular language. Then there exists an integer p > 1, called the pumping 
length, such that the following holds: Every string s in A, with |s| > p, can 
be written as s — xyz, such that 

1. y^e (i.e., \y\ > l), 

2. \xy\ < p, and 

3. for all i > 0, xy' l z € A. 

In words, the pumping lemma states that by replacing the portion y in s 
by zero or more copies of it, the resulting string is still in the language A. 

Proof. Let E be the alphabet of A. Since A is a regular language, there 
exists a DFA M = ( Q , E, S, q, F ) that accepts A. We define p to be the 
number of states in Q. 

Let s = sis 2 ... s n be an arbitrary string in A such that n > p. Define 
r\ = q, r 2 — 8(ri,si), r 3 — 8(r 2 ,s 2 ), ..., r n+1 — 5(r n ,s n ). Thus, when the 
DFA M reads the string s from left to right, it visits the states ri, r 2 ,..., r n+1 . 
Since s is a string in A, we know that r n+1 belongs to F. 

Consider the first p+ 1 states n, r 2 ,..., r p+ \ in this sequence. Since the 
number of states of M is equal to p, the pigeon hole principle implies that 
there must be a state that occurs twice in this sequence. That is, there are 
indices j and £ such that 1 < j < £ < p + 1 and r 3 = r(. 



We define x — sis 2 ... Sj_i, y — Sj ... st-i, and z — se ... s n . Since j < £, 
we have j/ / e, proving the first claim in the theorem. Since £ < p + 1, we 
have \xy\ = £ — 1 < p, proving the second claim in the theorem. To see that 
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the third claim also holds, recall that the string s = xyz is accepted by M. 
While reading x, M moves from the start state q to state r r While reading 
y, it moves from state rj to state re = rj, i.e., after having read y, M is again 
in state rj. While reading 0 , M moves from state rj to the accept state r n+ \. 
Therefore, the substring y can be repeated any number i > 0 of times, and 
the corresponding string xy l z will still be accepted by M. It follows that 
xy l z G A for all i > 0. ■ 


2.9.1 Applications of the pumping lemma 

First example 

Consider the language 

A = {0 n l n : n > 0}. 

We will prove by contradiction that A is not a regular language. 

Assume that A is a regular language. Let p> 1 be the pumping length, 
as given by the pumping lemma. Consider the string s = 0 P 1 P . It is clear 
that s G A and \s\ = 2p > p. Hence, by the pumping lemma, s can be 
written as s = xyz, where y ^ e, \xy\ < p, and xy l z G A for all i > 0. 

Observe that, since \xy\ < p, the string y contains only Os. Moreover, 
since y ^ e, y contains at least one 0. But now we are in trouble: None of 
the strings xy°z — xz, xy 2 z — xyyz, xy 3 z — xyyyz, ..., is contained in A. 
However, by the pumping lemma, all these strings must be in A. Hence, we 
have a contradiction and we conclude that A is not a regular language. 

Second example 

Consider the language 

A = {w G {0,1}* : the number of 0s in w equals the number of Is in w}. 

Again, we prove by contradiction that A is not a regular language. 

Assume that A is a regular language. Let p > 1 be the pumping length, 
as given by the pumping lemma. Consider the string s = 0 P 1 P . Then s G A 
and \s\ = 2 p > p. By the pumping lemma, s can be written as s = xyz, 
where y ^ e, \xy\ < p, and xy l z G A for all i > 0. 

Since \xy\ < p, the string y contains only 0s. Since y ^ e, y contains at 
least one 0. Therefore, the string xy 2 z — xyyz contains more 0s than Is, 
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which implies that this string is not contained in A. But, by the pumping 
lemma, this string is contained in A. This is a contradiction and, therefore, 
A is not a regular language. 

Third example 

Consider the language 


A — {ww : w G {0,1}*}. 

We prove by contradiction that A is not a regular language. 

Assume that A is a regular language. Let p > 1 be the pumping length, 
as given by the pumping lemma. Consider the string s = 0 P 10 P 1. Then s <G A 
and |s| = 2p T 2 > p. By the pumping lemma, s can be written as s = xyz , 
where y ^ e, \xy\ < p , and xy l z e A for all i > 0. 

Since \xy\ < p, the string y contains only Os. Since y ^ e, y contains at 
least one 0. Therefore, the string xy 2 z = xyyz is not contained in A. But, 
by the pumping lemma, this string is contained in A. This is a contradiction 
and, therefore, A is not a regular language. 

You should convince yourself that by choosing s — 0 2p (which is a string 
in A whose length is at least p), we do not obtain a contradiction. The reason 
is that the string y may have an even length. Thus, 0 2p is the “wrong” string 
for showing that A is not regular. By choosing s = 0 P 10 P 1, we do obtain 
a contradiction; thus, this is the “correct” string for showing that A is not 
regular. 

Fourth example 

Consider the language 


A = {0 m l ra : m > n > 0}. 

We prove by contradiction that A is not a regular language. 

Assume that A is a regular language. Let p > 1 be the pumping length, 
as given by the pumping lemma. Consider the string s = 0 P+1 l p . Then s G A 
and |s| = 2p + 1 > p. By the pumping lemma, s can be written as s = xyz , 
where y ^ e, \xy\ < p , and xy l z e A for all i > 0. 

Since \xy\ < p, the string y contains only 0s. Since y ^ e, y contains at 
least one 0. Consider the string xy°z — xz. The number of Is in this string 
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is equal to p , whereas the number of Os is at most equal to p. Therefore, the 
string xy°z is not contained in A. But, by the pumping lemma, this string 
is contained in A. This is a contradiction and, therefore, A is not a regular 
language. 

Fifth example 

Consider the language 

A — {l” 2 : n > 0}. 

We prove by contradiction that A is not a regular language. 

Assume that A is a regular language. Let p > 1 be the pumping length, 
as given by the pumping lemma. Consider the string s = l p . Then s G A 
and \s\ = p 2 > p. By the pumping lemma, s can be written as s = xyz, 
where y/e, \xy\ < p, and xy*z G A for all i > 0. 

Observe that 

|s| = \xyz\ =p 2 

and 

\xy 2 z\ = \xyyz\ = \xyz\ + \y\ = p 2 + \y\. 

Since \xy\ <p, we have \y\ < p. Since y ^ e, we have \y\ > 1. It follows that 
p 2 < \xy 2 z\ <p 2 +p< (p+ l) 2 . 

Hence, the length of the string xy 2 z is strictly between two consecutive 
squares. It follows that this length is not a square and, therefore, xy 2 z 
is not contained in A. But, by the pumping lemma, this string is contained 
in A. This is a contradiction and, therefore, A is not a regular language. 

Sixth example 

Consider the language 

A = { 1” : n is a prime number}. 

We prove by contradiction that A is not a regular language. 

Assume that A is a regular language. Let p > 1 be the pumping length, 
as given by the pumping lemma. Let n > p be a prime number, and consider 
the string s = l n . Then s G A and \s\ = n > p. By the pumping lemma, s 
can be written as s — xyz, where y ^ e, \xy\ < p, and xy l z G A for all % > 0. 
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Let k be the integer such that y = l k . Since y ^ e, we have k > 1. For 
each i > 0, n + {i — 1 )k is a prime number, because xy l z = i n +(* _1 ) fe g A. 
For i = n + 1, however, we have 

n+ (i — l)k = n + nk — n( 1 + k), 

which is not a prime number, because n > 2 and 1 + k > 2. This is a 
contradiction and, therefore, A is not a regular language. 

Seventh example 

Consider the language 

A = {w & {0,1}* : the number of occurrences of 01 in w is equal to 
the number of occurrences of 10 in w }. 

Since this language has the same flavor as the one in the second example, 
we may suspect that A is not a regular language. This is, however, not true: 
As we will show, A is a regular language. 

The key property is the following one: Let w be an arbitrary string in 
{0,1}*. Then 

the absolute value of the number of occurrences of 01 in w minus 
the number of occurrences of 10 in w is at most one. 

This property holds, because between any two consecutive occurrences of 
01, there must be exactly one occurrence of 10. Similarly, between any two 
consecutive occurrences of 10, there must be exactly one occurrence of 01. 

We will construct a DFA that accepts A. This DFA uses the following 
five states: 

• q: start state; no symbol has been read. 

• Qq] : the last symbol read was 1; in the part of the string read so far, the 
number of occurrences of 01 is one more than the number of occurrences 
of 10. 

• guy. the last symbol read was 0; in the part of the string read so far, the 
number of occurrences of 10 is one more than the number of occurrences 
of 01. 


Source: www.csitsansar.com 



2.10. Higman’s Theorem 


73 


• Qequai'- the last symbol read was 0; in the part of the string read so far, 
the number of occurrences of 01 is equal to the number of occurrences 
of 10. 

• q\ qua (- the last symbol read was 1; in the part of the string read so far, 
the number of occurrences of 01 is equal to the number of occurrences 


of 10. 


The set of accept states is equal to {q,qequal i^lquai}- The state diagram of 
the DFA is given below. 




In fact, the key property mentioned above implies that the language A 
consists of the empty string e and all non-empty binary strings that start 
and end with the same symbol. As a result, A is the language described by 
the regular expression 


e U 0 U 1 U 0(0 U 1)*0 U 1(0 U 1)*1. 


This gives an alternative proof for the fact that A is a regular language. 
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2.10 Higman’s Theorem 

Let E be a finite alphabet. For any two strings x and y in E*, we say that x 
is a subsequence of y, if x can be obtained by deleting zero or more symbols 
from y. For example, 10110 is a subsequence of 0010010101010001. For any 
language L C E*, we define 

SUBSEQ(L) := {x : there exists a y E L such that x is a subsequence of y}. 

That is, SUBSEQ(L) is the language consisting of the subsequences of all 
strings in L. In 1952, Higman proved the following result: 

Theorem 2.10.1 (Higman) For any finite alphabet E and for any lan¬ 
guage L C E*, the language SUBSEQ(L) is regular. 

2.10.1 Dickson’s Theorem 

Our proof of Higman’s Theorem will use a theorem that was proved in 1913 
by Dickson. 

Recall that N denotes the set of positive integers. Let n <G N. For any 
two points p — (pi,p 2 , ■ ■ ■ iPn) and q = (gi, q 2 , ..., q n ) in N n , we say that p is 
dominated by q, if p^ < qi for all i with 1 < i < n. 

Theorem 2.10.2 (Dickson) Let S C N n , and let M be the set consisting of 
all elements of S that are minimal in the relation “is dominated by”. Thus, 

M = {q e S : there is no p in S \ {g} such that p is dominated by g}. 

Then, the set M is finite. 

We will prove this theorem by induction on the dimension n. If n = 1, 
then either M = 0 (if S = 0) or M consists of exactly one element (if S 0). 
Therefore, the theorem holds if n — 1. Let n > 2 and assume the theorem 
holds for all subsets of N n_1 . Let S be a subset of N n and consider the set 
M of minimal elements in S'. If S = 0, then M = 0 and, thus, M is finite. 
Assume that S' ^ 0. We fix an arbitrary element q in M. If p e M \ {g}, 
then g is not dominated by p. Therefore, there exists an index i such that 
Pi < Qi ~ 1- It follows that 

M \ {g} C Q (N* _1 X [1, q { — 1] x N n_i ) . 
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For all i and k with 1 < i < n and 1 < k < g* — 1, we define 
Sik = {p e S : pi = k} 

and 

M ik = {p G M : Pi = k}. 

Then, 

< 2 - 4 ) 

1=1 


Lemma 2.10.3 is a subset of the set of all elements of S ik that are 
minimal in the relation “is dominated by”. 


Proof. Let p be an element of M lk , and assume that p is not minimal in 
Sik- Then there is an element r in S ik , such that r ^ p and r is dominated 
by p. Since p and r are both elements of S', it follows that p 0 M. This is a 
contradiction. ■ 


Since the set Sik is basically a subset of N n ~', it follows from the induction 
hypothesis that S, k contains finitely many minimal elements. This, combined 
with Lemma 2.10.3, implies that M\ k is a finite set. Thus, by (2.4), M \ {q} 
is the union of finitely many finite sets. Therefore, the set M is finite. 


2.10.2 Proof of Higman’s Theorem 

We give the proof of Theorem 2.10.1 for the case when E = {0,1}. If L = 0 
or SUBSEQ(L) = {0,1}*, then SUBSEQ(L) is obviously a regular language. 
Hence, we may assume that L is non-empty and SUBSEQ(L) is a proper 

subset of {0,1}*. _ 

We fix a string z of length at least two in the complement SUBSEQ(L) of 
the language SUBSEQ(L). Observe that this is possible, because SUBSEQ(L) 
is an infinite language. We insert 0s and Is into z, such that, in the result¬ 
ing string y, 0s and Is alternate. For example, if z = 0011101011, then 
z' = 01010101010101. Let n = \z'\ — 1, where \z'\ denotes the length of z'. 
Then, n > \z\ — 1 > 1. 

A (0, 1)-alternation in a binary string x is any occurrence of 01 or 10 in x. 
For example, the string 1101001 contains four (0,1 ^alternations. We define 

A = {x E {0,1}* : x has at most n many (0, l)-alternations}. 


Source: www.csitsansar.com 





76 


Chapter 2. Finite Automata and Regular Languages 


Lemma 2.10.4 SUBSEQ(L) C A. 

Proof. Let x G SUBSEQ(L) and assume that x ^ A. Then, x has at least 
n + 1 = \z'\ many (0, l)-alternations and, therefore, z' is a subsequence of x. 
In particular, z is a subsequence of x. Since x G SUBSEQ(L), it follows that 
0 G SUBSEQ(L), which is a contradiction. I 


Lemma 2.10.5 SUBSEQ{L ) = (An SUBSEQ(L)^ U A. 

Proof. Follows from Lemma 2.10.4. I 


Lemma 2.10.6 The language A is regular. 

Proof. The complement A of A is the language consisting of all binary 
strings with at least n + 1 many (0,1 ^alternations. If, for example, n = 3, 
then A is described by the regular expression 

(00*11*00*11*0(0 Ul)*) U (11*00*11*00*1(0 Ul)*). 

This should convince you that the claim is true for any value of n. ■ 

For any b G {0,1} and for any k > 0, we define A hk to be the language 
consisting of all binary strings that start with a b and have exactly k many 
(0, l)-alternations. Then, we have 


^l = {e}U 



Thus, if we define 

F bk = A bk n SUBSEQ(L), 

and use the fact that e 6 SUBSEQ(L) (which is true because L ^ 0), then 

1 n 

A n SUBSEQ(L) = Ulk- m 

6=0 k=0 
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For any b G {0,1} and for any k > 0, consider the relation “is a subse¬ 
quence of” on the language F bk . We define M hk to be the language consisting 
of all strings in F bk that are minimal in this relation. Thus, 

M bk = {xe F bk : there is no x' in F bk \ {a:} such that x' is a subsequence of x}. 

It is clear that 


F bk = [_J {y G F bk : x is a subsequence of y}. 

xeM bk 

If x G M bk , y G A bk , and a; is a subsequence of y, then y must be in 
SUBSEQ(L) and, therefore, y must be in F bk . To prove this, assume that 
y G SUBSEQ(L ). Then, x G SUBSEQ(L), contradicting the fact that 
x G M bk C F bk C SUBSEQ(L). It follows that 

F bk = {y G A bk : x is a subsequence of y}. (2.6) 

xeM bk 


Lemma 2.10.7 Let b G {0,1} and 0 < k < n, and let x be an element of 
M bk . Then, the language 

{ y G A bk : x is a subsequence of y} 


is regular. 

Proof. We will prove the claim by means of an example. Assume that b = 1, 
k = 3, and x — 11110001000. Then, the language 

{y G A bk : x is a subsequence of y} 

is described by the regular expression 

11111 * 0000 * 11 * 0000 *. 

This should convince you that the claim is true in general. ■ 


Lemma 2.10.8 For each b G {0,1} and each 0 < k < n, the set M bk is 
finite. 
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Proof. Again, we will prove the claim by means of an example. Assume 
that b = 1 and k = 3. Any string in F bk can be written as l a 0 6 l c 0 d , for some 
integers a,b,c,d > 1. Consider the function ip : F bk —> N 4 that is defined by 
ip{l a i) h Y'{) d ) = ( a,b,c,d ). Then, <p is an injective function, and the following 
is true, for any two strings x and x' in F bk : 

a; is a subsequence of x' if and only if <p(x) is dominated by ip{x'). 

It follows that the elements of M bk are in one-to-one correspondence with 
those elements of <p(F bk ) that are minimal in the relation “is dominated by”. 
The lemma thus follows from Dickson’s Theorem. I 

Now we can complete the proof of Higman’s Theorem: 

• It follows from (2.6) and Lemmas 2.10.7 and 2.10.8, that F bk is the 
union of finitely many regular languages. Therefore, by Theorem 2.3.1, 
F bk is a regular language. 

• It follows from (2.5) that AnSUBSEQ(L) is the union of finitely many 
regular languages. Therefore, again by Theorem 2.3.1, AnSUBSEQ(L) 
is a regular language. 

• Since A fl SUBSEQ(L) is regular and, by Lemma 2.10.6, A is regular, 
it follows from Lemma 2.10.5 that SUBSEQ(L) is the union of two reg¬ 
ular languages. Therefore, by Theorem 2.3.1, SUBSEQ(L) is a regular 
language. 

• Since SUBSEQ(L) is regular, it follows from Theorem 2.6.4 that the 
language SUBSEQ(L) is regular as well. 


Exercises 

2.1 For each of the following languages, construct a DFA that accepts the 
language. In all cases, the alphabet is {0,1}. 

1. {w : the length of w is divisible by three} 

2. {w : 110 is not a substring of w} 

3. {w : w contains at least five Is} 


Source: www.csitsansar.com 









Exercises 


79 


4. {w : w contains the substring 1011} 

5. {w : w contains at least two Is and at most two Os} 

6. {w : w contains an odd number of Is or exactly two Os} 

7. {w : w begins with 1 and ends with 0} 

8. {w : every odd position in w is 1} 

9. {w : w has length at least 3 and its third symbol is 0} 

10. {e, 0} 

2.2 For each of the following languages, construct an NFA, with the specified 
number of states, that accepts the language. In all cases, the alphabet is 
{ 0 , 1 }. 

1. The language {w : w ends with 10} with three states. 

2. The language {w : w contains the substring 1011} with five states. 

3. The language {w : w contains an odd number of Is or exactly two 0s} 
with six states. 

2.3 For each of the following languages, construct an NFA that accepts the 
language. In all cases, the alphabet is {0,1}. 

1. {w : w contains the substring 11001} 

2. {w : w has length at least 2 and does not end with 10} 

3. {w : w begins with 1 or ends with 0} 

2.4 Convert the following NFA to an equivalent DFA. 
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2.5 Convert the following NFA to an equivalent DFA. 



2.6 Convert the following NFA to an equivalent DFA. 



2.7 In the proof of Theorem 2.6.3, we introduced a new start state g 0 , which 
is also an accept state. Explain why the following is not a valid proof of 
Theorem 2.6.3: 

Let N = (Qi, E, Si,qi,Fi) be an NFA, such that A = L(N). Define the 
NFA M = (Qi, E, S, qi, F), where 

1. F = {gi} U Fi. 

2. 6 : Qi x E e — > V{Q\) is defined as follows: For any r e Q\ and for any 
a G E £ , 

{ 5i(r, a) if re Q i and r ^ F l , 

(r, a) if r G F 1 and a ^ e, 

5i(r, a) U {gi} if r e Fi and a = e. 


Source: www.csitsansar.com 






Exercises 


81 


Then L(M) = A*. 

2.8 Prove Theorem 2.6.4. 

2.9 Let A be a language over the alphabet E = {0,1} and let A be the 
complement of A. Thus, A is the language consisting of all binary strings 
that are not in A. 

Assume that A is a regular language. Let M — (Q, E, S, q, F ) be a non- 
deterministic finite automaton (NFA) that accepts A. 

Consider the NFA N = (Q, E, S, q, F ), where F = Q\F is the complement 
of F. Thus, N is obtained from M by turning all accept states into nonaccept 
states, and turning all nonaccept states into accept states. 

1. Is it true that the language accepted by N is equal to A? 

2. Assume now that M is a deterministic finite automaton (DFA) that 
accepts A. Define N as above; thus, turn all accept states into nonac¬ 
cept states, and turn all nonaccept states into accept states. Is it true 
that the language accepted by N is equal to A? 

2.10 Recall the alternative definition for the star of a language A that we 
gave just before Theorem 2.3.1. 

In Theorems 2.3.1 and 2.6.2, we have shown that the class of regular 
languages is closed under the union and concatenation operations. Since 
A* = IXo J ^ k -> why doesn’t this imply that the class of regular languages is 
closed under the star operation? 

2.11 Let A and B be two regular languages over the same alphabet E. Prove 
that the difference of A and B, i.e., the language 

A \ B = {w : w e A and w $ B} 


is a regular language. 

2.12 For each of the following regular expressions, give two strings that are 
members and two strings that are not members of the language described by 
the expression. The alphabet is E = {a, b}. 

1. a(ba)*b. 

2. (aU b)*a(aU b)*b(aU b)*a(aU b)*. 


Source: www.csitsansar.com 



82 


Chapter 2. Finite Automata and Regular Languages 


3. (a U ba U bb)(a U b)*. 

2.13 Give regular expressions describing the following languages. In all 
cases, the alphabet is {0,1}. 

1. {w : w contains at least three Is}. 

2. {w : w contains at least two Is and at most one 0}, 

3. {w : w contains an even number of Os and exactly two Is}. 

4. {w : w contains exactly two Os and at least two Is}. 

5. {w : w contains an even number of Os and each 0 is followed by at least one 1}. 

6. {w : every odd position in w is 1}. 

2.14 Convert each of the following regular expressions to an NFA. 

1. (0U 1)*000(0U1)* 

2. (((10)*(00))U10)* 

3. ((0U1)(11)*U0)* 

2.15 Convert the following DFA to a regular expression. 



2.16 Convert the following DFA to a regular expression. 
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2.17 Convert the following DFA to a regular expression. 



2.18 1. Let A be a non-empty regular language. Prove that there exists 
an NFA that accepts A and that has exactly one accept state. 

2. For any string w = W\W 2 ... w n , we denote by w R the string obtained 
by reading w backwards, i.e., w R = w n w n _ x ... w 2 W\. For any language 
A, we define A R to be the language obtained by reading all strings in 
A backwards, i.e., 

A r = {w R : w € A}. 

Let A be a non-empty regular language. Prove that the language A R 
is also regular. 

2.19 If n > 1 is an integer and w = ai<z 2 ... a n is a string, then for any i 
with 0 < i < n, the string aia 2 ... a, is called a proper prefix of w. (If i = 0, 
then aia 2 ... a* = e.) 

For any language L , we define MIN(L) to be the language 

MIN(L) = {w G L : no proper prefix of w belongs to L}. 

Prove the following claim: If L is a regular language, then MIN(L) is regular 
as well. 
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2.20 Use the pumping lemma to prove that the following languages are not 
regular. 

1. {a n b m c n+m : n > 0, m > 0}. 

2 ( a n 6 n c 2n • n > Q}. 

3. {a n b m a n : n > 0, m > 0}. 

4. {a 2 " : n > 0}. (Remark: a 2 " is the string consisting of 2 n many a’s.) 

5. {a n b m c k : n > 0, m > 0, k > 0, n 2 + m 2 = k 2 }. 

6. {uvu : u G {a, b}*, u ^ e, v G {a, 6}*}. 

2.21 Prove that the language 

{a m b n :m>0,n>0,m^n} 

is not regular. (Using the pumping lemma for this one is a bit tricky. You 
can avoid using the pumping lemma by combining results about the closure 
under regular operations.) 

2.22 1. Give an example of a regular language A and a non-regular lan¬ 
guage B for which A C B. 

2. Give an example of a non-regular language A and a regular language 
B for which AC B. 

2.23 Let A be a language consisting of finitely many strings. 

1. Prove that A is a regular language. 

2. Let n be the maximum length of any string in A. Prove that every 
deterministic finite automaton (DFA) that accepts A has at least n+1 
states. {Hint: How is the pumping length chosen in the proof of the 
pumping lemma?) 

2.24 Let L be a regular language, let M be a DFA whose language is equal 
to L, and let p be the number of states of M. Prove that L ^ 0 if and only 
if L contains a string of length less than p. 
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2.25 Let L be a regular language, let M be a DFA whose language is equal 
to L, and let p be the number of states of M. Prove that L is an infinite 
language if and only if L contains a string w with p < \w\ < 2p — 1. 

2.26 Let E be a non-empty alphabet, and let L be a language over E, i.e., 
LCE*. We define a binary relation R L on E* x E*, in the following way: 
For any two strings u and u! in E*, 

uRl,u' if and only if (Vu G E* : ™ G L O u'v 6 L) . 

Prove that Rl is an equivalence relation. 


2.27 Let E = {0,1}, let 


L = {w G E* : |rc| is odd}, 
and consider the relation R L defined in Exercise 2.26. 
1. Prove that for any two strings u and u' in E*, 

uR l u' \u\ — M is even. 


2. Determine all equivalence classes of the relation R L . 

2.28 Let E be a non-empty alphabet, and let L be a language over E, i.e., 
L C E*. Recall the equivalence relation Rl that was defined in Exercise 2.26. 

1. Assume that L is a regular language, and let M — (Q, E, S, q 0 , F ) be 
a DFA that accepts L. Let u and v! be strings in E*. Let q be the 
state reached, when following the path in the state diagram of M, that 
starts in q 0 and that is obtained by reading the string u. Similarly, let 
q' be the state reached, when following the path in the state diagram 
of M, that starts in q 0 and that is obtained by reading the string v!. 

Prove the following: If q = q', then uR L u'. 

2. Prove the following claim: If L is a regular language, then the equiva¬ 
lence relation R L has a finite number of equivalence classes. 
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2.29 Let L be the language defined by 

L = {uu R : u €E {0,1}*}. 

In words, a string is in L if and only if its length is even, and the second half 
is the reverse of the first half. Consider the equivalence relation R L that was 
defined in Exercise 2.26. 

1. Let m and n be two distinct positive integers and consider the two 
strings u = 0 m l and v! = 0”1. Prove that ->(uRlu'). 

2. Prove that L is not a regular language, without using the pumping 
lemma. 

3. Use the pumping lemma to prove that L is not a regular language. 

2.30 In this exercise, we will show that the converse of the pumping lemma 
does, in general, not hold. Consider the language 

A = {a m b n c n : m > 1, n > 0} U {b n c k : n > 0, k > 0}. 

1. Show that A satisfies the conclusion of the pumping lemma for p = 1. 
Thus, show that every string s in A whose length is at least p can be 
written as s = xyz, such that y ^ e, \xy\ < p, and xy l z <G A for all 
i > 0. 

2. Consider the equivalence relation Ra that was defined in Exercise 2.26. 
Let n and n' be two distinct non-negative integers and consider the two 
strings u = ab n and u! = ab n . Prove that ->(uRau'). 

3. Prove that A is not a regular language. 
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Chapter 3 

Context-Free Languages 


In this chapter, we introduce the class of context-free languages. As we 
will see, this class contains all regular languages, as well as some nonregular 
languages such as {0 n l n : n > ()}. 

The class of context-free languages consists of languages that have some 
sort of recursive structure. We will see two equivalent methods to obtain this 
class. We start with context-free grammars, which are used for defining the 
syntax of programming languages and their compilation. Then we introduce 
the notion of (nondeterministic) pushdown automata, and show that these 
automata have the same power as context-free grammars. 


3.1 Context-free grammars 

We start with an example. Consider the following five (substitution) rules: 

S -)• AB 
A a 
A —y aA 
B -* b 
B -)■ bB 

Here, S, A, and B are variables , S is the start variable , and a and b are 
terminals. We use these rules to derive strings consisting of terminals (i.e., 
elements of {a, &}*), in the following manner: 

1. Initialize the current string to be the string consisting of the start 
variable S. 
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2. Take any variable in the current string and take any rule that has this 
variable on the left-hand side. Then, in the current string, replace this 
variable by the right-hand side of the rule. 

3. Repeat 2. until the current string only contains terminals. 

For example, the string aaaabb can be derived in the following way: 

S =$■ AB 
=> aAB 
=> aAbB 
=>- aaAbB 
=> aaaAbB 
=> aaaabB 
=>■ aaaabb 

This derivation can also be represented using a parse tree , as in the figure 
below: 


s 



The five rules in this example constitute a context-free grammar. The 
language of this grammar is the set of all strings that 
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• can be derived from the start variable and 

• only contain terminals. 

For this example, the language is 

{a m b n : m > 1, n > 1}, 

because every string of the form a m b n , for some m > 1 and n > 1, can be 
derived from the start variable, whereas no other string over the alphabet 
{a,b} can be derived from the start variable. 

Definition 3.1.1 A context-free grammar is a 4-tuple G = (V, E, 12, S), 
where 

1. V is a finite set, whose elements are called variables , 

2. E is a finite set, whose elements are called terminals , 

3. yns = 0, 

4. S is an element of V; it is called the start variable, 

5. R is a finite set, whose elements are called rules. Each rule has the 
form A —>- w, where A e V and w e (V U E)*. 

In our example, we have V = {.S', A, B}, E = {a, b}, and 

R={S -> AB,A^ a,A^ aA,B ^b,B -> bB}. 

Definition 3.1.2 Let G = ( V, E, R, S) be a context-free grammar. Let A be 
an element in V and let u, v, and w be strings in (V U E)* such that A —> w 
is a rule in R. We say that the string uwv can be derived in one step from 
the string uAv, and write this as 

uAv =>• uwv. 

In other words, by applying the rule A —> w to the string uAv, we obtain 
the string uwv. In our example, we see that aaAbb => aaaAbb. 

Definition 3.1.3 Let G = (V, E, R, S) be a context-free grammar. Let u 
and v be strings in ( V UE)*. We say that v can be derived from u , and write 
this as u =r- v, if one of the following two conditions holds: 
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1. u = v or 

2. there exist an integer k > 2 and a sequence iq, u 2 , ..., u k of strings in 
(7 U S)*, such that 

(a) u — ui, 

(b) v — u k , and 

(c) U\ =>• U 2 => • • • => Uk- 

In other words, by starting with the string u and applying rules zero or 
more times, we obtain the string v. In our example, we see that aaAbB =>- 
aaaabbbB. 

Definition 3.1.4 Let G — (V, £ ,R,S) be a context-free grammar. The 
language of G is defined to be the set of all strings in E* that can be derived 
from the start variable S: 

L(G) = {weZ* :S^w}. 

Definition 3.1.5 A language L is called context-free , if there exists a context- 
free grammar G such that L(G) = L. 

3.2 Examples of context-free grammars 

3.2.1 Properly nested parentheses 

Consider the context-free grammar G — (V, £, i?, S'), where V = {A}, £ = 
{a, b }, and 

R = {S -)■ e, S -)• aSb, S -»• S3}. 

We write the three rules in R as 

S -4 e|aS'6|5'i5', 

where you can think of “|” as being a short-hand for “or”. 
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By applying the rules in R , starting with the start variable S, we obtain, 
for example, 


S =*• SS 
=£- aSbS 
=► aSbSS 
=> aSSbSS 
=> aaSbSbSS 
=> aabSbSS 
=>■ aabbSS 
=>■ aabbaSbS 
=> aabbabS 
=> aabbabaSb 
=>■ aabbabab 

What is the language L(G) of this context-free grammar G7 If we think 
of a as being a left-parenthesis “(”, and of b as being a right-parenthesis “)”, 
then L{G) is the language consisting of all strings of properly nested paren¬ 
theses. Here is the explanation: Any string of properly nested parentheses is 
either 


• empty (which we derive from S by the rule S' — > e), 

• consists of a left-parenthesis, followed by an arbitrary string of properly 
nested parentheses, followed by a right-parenthesis (these are derived 
from S by first applying the rule S —>• aSb), or 

• consists of an arbitrary string of properly nested parentheses, followed 
by an arbitrary string of properly nested parentheses (these are derived 
from S by first applying the rule S —>• SS). 

3 . 2.2 A context-free grammar for a nonregular lan¬ 
guage 

Consider the language L x = {0 n l n : n > 0}. We have seen in Section 2.9.1 
that Li is not a regular language. We claim that Li is a context-free language. 
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In order to prove this claim, we have to construct a context-free grammar 
G x such that L(Gi) = L x . 

Observe that any string in L\ is either 

• empty or 

• consists of a 0, followed by an arbitrary string in Li, followed by a 1. 

This leads to the context-free grammar G\ = (Vj, E, R x , Si), where V x = 

E = {0,1}, and R\ consists of the rules 

5! -)• e|0S x l. 

Hence, R x = {Si -)• e, S x ->• OSil}. 

To derive the string 0”l n from the start variable Si, we do the following: 

• Starting with Si, apply the rule S x —y Os'll exactly n times. This gives 
the string 0 n Si 1". 

• Apply the rule Si —>• e. This gives the string 0 n l n . 

It is not difficult to see that these are the only strings that can be derived 
from the start variable Si. Thus, L{G X ) = L x . 

In a symmetric way, we see that the context-free grammar G 2 = (V 2 , E, R 2 , S 2 ), 
where V 2 = {S 2 }, E = {0,1}, and R 2 consists of the rules 

S 2 —y e|lS20, 

has the property that L(G 2 ) = L 2 , where L 2 = {l n 0 n : n > ()}. Thus, L 2 is 
a context-free language. 

Define L = L x U L 2 , i.e., 


L = {0 n l n : n > 0} U {l n 0 n : n > 0}. 

The context-free grammar G = (V, E ,R,S), where V = {S, Si,S 2 }, E = 
{0,1}, and R consists of the rules 

S Si|S 2 

51 ->• e|0Sil 

5 2 —y e|lS 2 0, 

has the property that L(G) = L. Hence, L is a context-free language. 
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3.2.3 A context-free grammar for the complement of 
a nonregular language 

Let L be the (nonregular) language L = {0 n l n : n > ()}. We want to prove 
that the complement L of L is a context-free language. Hence, we want to 
construct a context-free grammar G whose language is equal to L. Observe 
that a binary string w is in L if and only if 

1. w — 0 m l n , for some integers m and n with 0 < m < n, or 

2. w = 0 m l n , for some integers m and n with 0 < n < m, or 

3. w contains 10 as a substring. 

Thus, we can write L as the union of the languages of all strings of type 1., 
type 2., and type 3. 

Any string of type 1. is either 

• the string 1, 

• consists of a string of type 1., followed by one 1, or 

• consists of one 0, followed by an arbitrary string of type 1., followed by 
one 1. 

Thus, using the rules 

Si ->• llSjllOSil, 

we can derive, from Si, all strings of type 1. 

Similarly, using the rules 


S2 —>■ OIOS2IOS2I, 

we can derive, from S 2 , all strings of type 2. 

Any string of type 3. 

• consists of an arbitrary binary string, followed by the string 10, followed 
by an arbitrary binary string. 

Using the rules 

X e|0X|lX, 
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we can derive, from X, all binary strings. Thus, by combining these with 
the rule 

S 3 X10X, 

we can derive, from S 3 , all strings of type 3. 

We arrive at the context-free grammar G = (V, E, R. S), where V = 
{S, Si, S 2 , S 3 , X}, E = {0,1}, and R consists of the rules 

S SilSslSa 

51 ->• llSiljoSjl 

52 —y 0I0S2I0S21 

5 3 xiox 

X e\QX\lX 

To summarize, we have 

51 =>■ 0 m l n , for all integers m and n with 0 < m < n, 

5 2 =>■ 0 m l n , for all integers m and n with 0 < n < m, 

X u, for each string u in {0,1}*, 

and 


S 3 => w, for every binary string w that contains 10 as a substring. 
From these observations, it follows that that L(G) = L. 

3.2.4 A context-free grammar that verifies addition 

Consider the language 


L = {a n b m c n+m : n > 0, m > 0}. 

Using the pumping lemma for regular languages (Theorem 2.9.1), it can 
be shown that L is not a regular language. We will construct a context- 
free grammar G whose language is equal to L, thereby proving that L is a 
context-free language. 

First observe that e £ i. Therefore, we will take S —> e to be one of the 
rules in the grammar. 

Let us see how we can derive all strings in L from the start variable S: 
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1. Every time we add an a, we also add a c. In this way, we obtain all 
strings of the form a n c n , where n> 0. 

2. Given a string of the form a n c n , we start adding bs. Every time we add 
a b , we also add a c. Observe that every b has to be added between 
the as and the cs. Therefore, we use a variable B as a “pointer” to 
the position in the current string where a b can be added: Instead of 
deriving a n c n from S, we derive the string a n Bc n . Then, from B, we 
derive all strings of the form b m c m , where m > 0. 

We obtain the context-free grammar G = (V, E, R, S ), where V = {S', A, B}, 
E = {a, b,c}, and R consists of the rules 

S e\A 

A —> e\aAc\B 
B ej bBc 

The facts that 

• A =>■ a n Bc n , for every n > 0, 

• B b m c m , for every m > 0, 

imply that the following strings can be derived from the start variable S: 

• S =>- a n Bc n a n b m c m c n — a n b m c n+m , for all n > 0 and m > 0. 

In fact, no other strings in {a, b, c}* can be derived from S. Therefore, we 
have L{G) = L. Since 

S^A^B^e, 

we can simplify this grammar G , by eliminating the rules S —> e and A —> e. 
This gives the context-free grammar G' = ( V. , E, R', S), where V = {S, A, B}, 

E = {a, b,c}, and R! consists of the rules 

S -)■ A 
A —>■ a^4c|R 
B —y e| bBc 

Finally, observe that we do not need S; instead, we can use A as start 
variable. This gives our final context-free grammar G" = (V, E, R", A ), where 
V = {A, B}, E = {a, b, c}, and R" consists of the rules 

A aAc\B 
B e\bBc 
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3.3 Regular languages are context-free 

We mentioned already that the class of context-free languages includes the 
class of regular languages. In this section, we will prove this claim. 

Theorem 3.3.1 Let E be an alphabet and let L CS* be a regular language. 
Then L is a context-free language. 

Proof. Since L is a regular language, there exists a deterministic finite 
automaton M — ( Q , E, 5, q, F) that accepts L. 

To prove that L is context-free, we have to define a context-free grammar 
G = (V,E ,R,S), such that L = L(M) = L(G). Thus, G must have the 
following property: For every string w E E*, 

w E L(M ) if and only if w € L(G), 

which can be reformulated as 

M accepts w if and only if S =4> w. 

We will define the context-free grammar G in such a way that the following 
correspondence holds for any string w = W\W 2 ... w n : 

• Assume that M is in state A just after it has read the substring 
W\W 2 ■ ■ -Wi- 

• Then in the context-free grammar G, we have S => w±w 2 ■ ■ ■ WiA. 

In the next step, M reads the symbol ruj+i and switches from state A to, 
say, state B ; thus, S(A,w i+ 1 ) — B. In order to guarantee that the above 
correspondence still holds, we have to add the rule A —>• w i+1 B to G. 

Consider the moment when M has read the entire string w. Let A be the 
state M is in at that moment. By the above correspondence, we have 

S w\U )2 ■ ■ ■ w n A = wA. 

Recall that G must have the property that 

M accepts w if and only if S =>• w, 

which is equivalent to 

A E F if and only if S =» w. 
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We guarantee this property by adding to G the rule A — > e for every accept 
state A of M. 

We are now ready to give the formal definition of the context-free gram¬ 
mar G = (V,E,R,S): 

• V = Q, i.e., the variables of G are the states of M. 

• S = q, i.e., the start variable of G is the start state of M. 

• R consists of the rules 

A — > aB, where A E Q, a E £, B E Q, and 8{A, a) = B , 


and 


A -4 e, where A E F. 


In words, 

• every transition S(A, a) = B of M (i.e., when M is in the state A and 
reads the symbol a, it switches to the state B) corresponds to a rule 
A —> aB in the grammar G, 

• every accept state A of M corresponds to a rule A —> e in the grammar 

G. 

We claim that L(G) = L. In order to prove this, we have to show that 
L(G) C L and L C L(G). 

We prove that L C L(G). Let w = w±w 2 ■ ■ ■ w n be an arbitrary string 
in L. When the finite automaton M reads the string w, it visits the states 
r 0 , Ti,.. ■, r n , where 

• r 0 = q, and 

• r i+ 1 = 5(ri, w i+ i) for i — 0,1 ,..., n — 1. 

Since w 6 L = L(M), we know that r n 6 F. 

It follows from the way we defined the grammar G that 

• for each i = 0,1 ,..., n — 1, r* w i+ ir i+ i is a rule in R , and 

• r n —> e is a rule in R. 
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Therefore, we have 

S = q = r 0 =>• wiri =>■ w\w 2 r 2 =>■ ... =>■ =>■ w\w 2 .. .w n — w. 

This proves that w e L(G). 

The proof of the claim that L(G) C L is left as an exercise. ■ 

In Sections 2.9.1 and 3.2.2, we have seen that the language {0 n l n : n > 
0} is not regular, but context-free. Therefore, the class of all context-free 
languages properly contains the class of regular languages. 

3.3.1 An example 

Let L be the language defined as 

L = {w E {0,1}* : 101 is a substring of w}. 

In Section 2.2.2, we have seen that L is a regular language. In that section, 
we constructed the following deterministic finite automaton M that accepts 
L (we have renamed the states): 



We apply the construction given in the proof of Theorem 3.3.1 to convert 
M to a context-free grammar G whose language is equal to L. According 
to this construction, we have G — (V f £,. R, S), where V = {S,A,B,C}, 
E = {0,1}, the start variable S is the start state of M, and R consists of the 
rules 

S 0S|1A 

A ->■ QB\1A 
B 0S|1C 

C 0C\lC\e 
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Consider the string 010011011, which is an element of L. When the finite 
automaton M reads this string, it visits the states 

5, 5, A, 5, 5, A, A, 5, C, C. 

In the grammar G, this corresponds to the derivation 
S => OS 
=> 01A 
=> 0105 
01005 
=» 010014 
=► 0100114 
=>- 01001105 
=► 01001101(7 
=>- 010011011(7 
=>- 010011011 . 

Hence, 

S 010011011, 

implying that the string 010011011 is in the language L(G) of the context-free 
grammar G. 

The string 10011 is not in the language L. When the finite automaton 
M reads this string, it visits the states 

S, A, 5, S, A, A, 

i.e., after the string has been read, M is in the non-accept state A. In the 
grammar G, reading the string 10011 corresponds to the derivation 

S => 1A 
=> 105 
=» 1005 
=>• 10014 
=> 100114. 

Since A is not an accept state in M, the grammar G does not contain the 
rule 4 —> e. This implies that the string 10011 cannot be derived from the 
start variable 5. Thus, 10011 is not in the language L(G) of G. 
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3.4 Chomsky normal form 

The rules in a context-free grammar G = (V, E, R, S) are of the form 
A —> w, 

where A is a variable and w is a string over the alphabet V U E. In this 
section, we show that every context-free grammar G can be converted to a 
context-free grammar G ', such that L(G) = L(G'), and the rules of G 1 are of 
a restricted form, as specified in the following definition: 

Definition 3.4.1 A context-free grammar G = (V, E, R , S) is said to be in 
Chomsky normal form , if every rule in R has one of the following three forms: 

1. A —> BC, where A, B, and C are elements of V, B ^ S, and C ^ S. 

2. A —y a, where A is an element of V and a is an element of E. 

3. S —> e, where S is the start variable. 

You should convince yourself that, for such a grammar, R contains the 
rule S —> e if and only if e & L(G). 

Theorem 3.4.2 Let E be an alphabet and let L C E* be a context-free lan¬ 
guage. There exists a context-free grammar in Chomsky normal form, whose 
language is L. 

Proof. Since L is a context-free language, there exists a context-free gram¬ 
mar G — (V, E, R, S), such that L(G) = L. We will transform G into a 
grammar that is in Chomsky normal form and whose language is equal to 
L(G). The transformation consists of five steps. 

Step 1: Eliminate the start variable from the right-hand side of the rules. 

We define G\ — {V\ , E, R.\, Sf), where S\ is the start variable (which is a 
new variable), V\ = V U {.SY}, and R.\ = RG {Si —> S'}. This grammar has 
the property that 

• the start variable Si does not occur on the right-hand side of any rule 
in Ri, and 

• L(Gi) = L(G). 
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Step 2: An e-rule is a rule that is of the form A —> e, where A is a variable 
that is not equal to the start variable. In the second step, we eliminate all 
e-rules from G\. 

We consider all e-rules, one after another. Let A —> e be one such rule, 
where A e Vi and We modify G\ as follows: 

1. Remove the rule A —>■ e from the current set R\. 

2. For each rule in the current set R\ that is of the form 

(a) B —> A, add the rule B —y e to Ri, unless this rule has already 
been deleted from R.\ ; observe that in this way, we replace the two- 
step derivation B =>- A =>- e by the one-step derivation B =>- e; 

(b) B —>• uAv (where u and v are strings that are not both empty), 
add the rule B —> uv to Ri, observe that in this way, we replace 
the two-step derivation B =>- uAv =>- uv by the one-step derivation 
B =>- uv; 

(c) B —> uAvAw (where u, v, and w are strings), add the rules B —> 
uvw, B —>• uAvw, and B —>• uvAw to R\; if u = v = w = e and 
the rule B —> e has already been deleted from R,\ , then we do not 
add the rule B —> e; 

(d) treat rules in which A occurs more than twice on the right-hand 
side in a similar fashion. 

We repeat this process until all e-rules have been eliminated. Let R 2 
be the set of rules, after all e-rules have been eliminated. We define G 2 = 
(V 2 , E, R 2 , S 2 ), where V 2 = V\ and S 2 = S\. This grammar has the property 
that 


• the start variable S 2 does not occur on the right-hand side of any rule 
in R 2 , 

• R 2 does not contain any e-rule (it may contain the rule S 2 —> e), and 

• L{G 2 ) = LiGr) - L{G). 


Step 3: A unit-rule is a rule that is of the form A —>• R, where A and B are 
variables. In the third step, we eliminate all unit-rules from G 2 . 
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We consider all unit-rules, one after another. Let A —> B be one such 
rule, where A and B are elements of V 2 . We know that B ^ S 2 . We modify 
G 2 as follows: 

1. Remove the rule A —» B from the current set R 2 . 

2. For each rule in the current set R 2 that is of the form B —>• u, where 
« G (y 2 U £)*, add the rule A —> u to the current set R 2 , unless this is 
a unit-rule that has already been eliminated. 

Observe that in this way, we replace the two-step derivation A =>- B =>■ 
u by the one-step derivation A =>- u. 

We repeat this process until all unit-rules have been eliminated. Let 
R 3 be the set of rules, after all unit-rules have been eliminated. We define 
G :i = (V 3 , E, ft’s, S 3 ), where V :i = V 2 and S 3 = S 2 . This grammar has the 
property that 

• the start variable S 3 does not occur on the right-hand side of any rule 
in R 3 , 

• R 3 does not contain any e-rule (it may contain the rule £3 —> e), 

• R 3 does not contain any unit-rule, and 


. L(G 3 )=L(G 2 ) = L(G 1 ) = L(G). 


Step 4: Eliminate all rules having more than two symbols on the right-hand 
side. 

For each rule in the current set R 3 that is of the form A —> U\U 2 
where k > 3 and each u t is an element of V 3 U E, we modify G 3 as follows: 

1. Remove the rule A —> U\U 2 ... u k from the current set R 3 . 

2 . Add the following rules to the current set R 3 : 


A 

Ax 

A 2 


—> u\Ax 
—y u 2 A 2 
u 3 A 3 


Ak -3 —y Uk-2Ak-2 
Ak-2 ► U k -lU k 


Source: www.csitsansar.com 



3.4. Chomsky normal form 


103 


where A 4 , A 2 ,..., A k _ 2 are new variables that are added to the current 
set V :j . 

Observe that in this way, we replace the one-step derivation A =>■ 
u 4 u 2 ■ ■ ■ u k by the (k — l)-step derivation 

A =>• uiAi =>• U\U 2 A 2 =>- ... => U\U 2 ... u k _ 2 A k _ 2 =>- uiu 2 ...u k . 


Let R 4 be the set of rules, and let V4 be the set of variables, after all rules 
with more than two symbols on the right-hand side have been eliminated. We 
define G 4 = (V4, E, R 4 , S 4 ), where S 4 = S3. This grammar has the property 
that 

• the start variable S 4 does not occur on the right-hand side of any rule 
in i? 4 , 

• R 4 does not contain any e-rule (it may contain the rule S 4 —^ e), 

• R 4 does not contain any unit-rule, 

• R 4 does not contain any rule with more than two symbols on the right- 
hand side, and 

• L(G 4 ) = L(G 3 ) = L(G 2 ) = L(Gi) = L[G). 

Step 5 : Eliminate all rules of the form A —>• u 4 u 2 , where u\ and u 2 are not 
both variables. 

For each rule in the current set R 4 that is of the form A —> u 4 u 2 , where 
Ui and u 2 are elements of V 4 U E, but u x and u 2 are not both contained in 
V4, we modify G3 as follows: 

1. If rti e E and u 2 &V 4 , then replace the rule A —>• u 4 u 2 in the current 

set R 4 by the two rules A —>• U 4 u 2 and U\ —>■ U\ , where U 4 is a new 

variable that is added to the current set V 4 . 

Observe that in this way, we replace the one-step derivation A => u 4 u 2 
by the two-step derivation A => U±u 2 => U\U 2 . 

2. If ui G V 4 and u 2 e E, then replace the rule A — y u 4 u 2 in the current 

set R 4 by the two rules A —>• u 4 U 2 and U 2 —>■ u 2 , where U 2 is a new 

variable that is added to the current set V 4 . 

Observe that in this way, we replace the one-step derivation A => u 4 u 2 
by the two-step derivation A => UiU 2 u 4 u 2 . 
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3. If ui e E, u 2 £ E, and ^ u 2 , then replace the rule A —y u 4 u 2 in the 
current set R, 4 by the three rules A UiU 2 , U { —>• u 4 , and U 2 —> u 2 , 
where U\ and U 2 are new variables that are added to the current set 

V 4 . 

Observe that in this way, we replace the one-step derivation A =>- u\u 2 
by the three-step derivation A =>• U\U 2 =>- U\U 2 =>- U\U 2 . 

4. If ui e E, u 2 G E, and u 4 — u 2 , then replace the rule A —> u 4 u 2 = u 4 ui 
in the current set R 4 by the two rules A —> U\U\ and U\ —>■ u 4 , where 
Ui is a new variable that is added to the current set V 4 . 

Observe that in this way, we replace the one-step derivation A =>- 
u 4 u 2 = U\U\ by the three-step derivation A^U\U\ => uRIi => U\U\. 

Let R 5 be the set of rules, and let V 5 be the set of variables, after Step 5 
has been completed. We define G 5 = (Vs,E ,R 5 ,S 5 ), where S 5 — S 4 . This 
grammar has the property that 

• the start variable S 5 does not occur on the right-hand side of any rule 
in R 5 , 

• R 5 does not contain any e-rule (it may contain the rule S 5 —t e), 

• R 5 does not contain any unit-rule, 

• i ?5 does not contain any rule with more than two symbols on the right- 
hand side, 

• R 5 does not contain any rule of the form A —>• u 4 u 2 , where U\ and u 2 
are not both variables of V 5 , and 


• L(G 5 ) = L{G 4 ) = L(G 3 ) = L{G 2 ) = L(G0 = L(G). 

Since the grammar G 5 is in Chomsky normal form, the proof is complete. ■ 
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3.4.1 An example 

Consider the context-free grammar G = (V, E, it*, A), where V = {A,B}, 
E = {0,1}, A is the start variable, and R consists of the rules 

A ->• BAB\B\e 
B ->• 00|e 

We apply the construction given in the proof of Theorem 3.4.2 to convert 
this grammar to a context-free grammar in Chomsky normal form whose 
language is the same as that of G. Throughout the construction, upper case 
letters will denote variables. 

Step 1: Eliminate the start variable from the right-hand side of the rules. 

We introduce a new start variable S , and add the rule S —> A. This gives 
the following grammar: 

S ->• A 
A BAB\B\e 
B -»■ 00|e 

Step 2: Eliminate all e-rules. 

We take the e-rule A —> e, and remove it. Then we consider all rules that 
contain A on the right-hand side. There are two such rules: 

• S — > A\ we add the rule S — > e; 

• A —> BAB\ we add the rule A —> BB. 

This gives the following grammar: 

S -> A\e 
A BAB\B\BB 

B 00|e 

We take the e-rule B —> e, and remove it. Then we consider all rules that 
contain B on the right-hand side. There are three such rules: 

• A — >■ BAB‘, we add the rules A — > AB, A — > BA, and A —>• A\ 

• A —>• B\ we do not add the rule A —>• e, because it has aheady been 
removed; 
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• A —>• BB: we add the rule A —>• B, but not the rule A —> e (because it 
has already been removed). 

At this moment, we have the following grammar: 

S ->• A\e 

A ->■ BAB\B\BB\AB\BA\A 
B 00 

Since all e-rules have been eliminated, this completes Step 2. (Observe that 
the rule S —> e is allowed, because S is the start variable.) 

Step 3: Eliminate all unit-rules. 

We take the unit-rule A —> A. We can remove this rule, without adding 
any new rule. At this moment, we have the following grammar: 

S A\e 

A -4 BAB\B\BB\AB\BA 
B -+ 00 

We take the unit-rule S —>• A, remove it, and add the rules 

S ^ BAB\B\BB\AB\BA. 

This gives the following grammar: 

S ->■ e\BAB\B\BB\AB\BA 
A BAB\B\BB\AB\BA 

B ^ 00 

We take the unit-rule S —> B, remove it, and add the rule S —>• 00. This 
gives the following grammar: 

S ->■ e\BAB\BB\AB\BA\00 
A ->■ SAS|S|SS|AS|SA 
S ^ 00 

We take the unit-rule A —> B, remove it, and add the rule A —> 00. This 
gives the following grammar: 

S ->• e\BAB\BB\AB\BA\00 
A ->• BAB\BB\AB\BA\m 
B 00 
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Since all unit-rules have been eliminated, this concludes Step 3. 

Step 4: Eliminate all rules having more than two symbols on the right-hand 
side. There are two such rules: 

• We take the rule S BAB, remove it, and add the rules S —>• BA\ 
and A\ — y AB. 

• We take the rule A —> BAB , remove it, and add the rules A —> BA 2 
and A 2 —> AB. 

This gives the following grammar: 

S -+ e\BB\AB\BA\OQ\BA 1 
A -> BB\AB\BA\00\BA 2 
B ^ 00 
-► AB 
A 2 -»• AB 

Step 4 is now completed. 

Step 5: Eliminate all rules, whose right-hand side contains exactly two 
symbols, which are not both variables. There are three such rules: 

• We replace the rule S —>• 00 by the rules S —> A 3 A 3 and A 3 —>• 0. 

• We replace the rule A —>• 00 by the rules A — > A 4 A 4 and A 4 —>• 0. 

• We replace the rule B —>• 00 by the rules B — > A 5 A 5 and A 5 — > 0. 

This gives the following grammar, which is in Chomsky normal form: 

S -+ elBBlABlBAlBA^AsAs 
A BB\AB\BA\BA 2 \A 4 A 4 

B —>• A 5 A 5 
A 1 AB 

A 2 AB 

A 3 —>• 0 
A 4 —y 0 
A5 y 0 
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3.5 Pushdown automata 

In this section, we introduce nondeterministic pushdown automata. As we 
will see, the class of languages that can be accepted by these automata is 
exactly the class of context-free languages. 

We start with an informal description of a deterministic pushdown au¬ 
tomaton. Such an automaton consists of the following, see also Figure 3.1. 

1. There is a tape which is divided into cells. Each cell stores a symbol 
belonging to a finite set E, called the tape alphabet. There is a special 
symbol □ that is not contained in E; this symbol is called the blank 
symbol. If a cell contains □, then this means that the cell is actually 
empty. 

2. There is a tape head which can move along the tape, one cell to the 
right per move. This tape head can also read the cell it currently scans. 

3. There is a stack containing symbols from a finite set T, called the stack 
alphabet. This set contains a special symbol $. 

4. There is a stack head which can read the top symbol of the stack. This 
head can also pop the top symbol, and it can push symbols of T onto 
the stack. 

5. There is a state control , which can be in any one of a finite number 
of states. The set of states is denoted by Q. The set Q contains one 
special state q, called the start state. 

The input for a pushdown automaton is a string in E*. This input string 
is stored on the tape of the pushdown automaton and, initially, the tape head 
is on the leftmost symbol of the input string. Initially, the stack only contains 
the special symbol $, and the pushdown automaton is in the start state q. 
In one computation step, the pushdown automaton does the following: 

1. Assume that the pushdown automaton is currently in state r. Let a be 
the symbol of E that is read by the tape head, and let A be the symbol 
of T that is on top of the stack. 

2. Depending on the current state r, the tape symbol a, and the stack 
symbol A, 
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|a|a|6|a|6|fe|fl|fe|a|fr |~5] tape 


state control 


A 

B_ 

A_ 

A_ 

$ 


stack 


Figure 3.1: A pushdown automaton. 


(a) the pushdown automaton switches to a state r' of Q (which may 
be equal to r), 

(b) the tape head either moves one cell to the right or stays at the 
current cell, and 

(c) the top symbol A is replaced by a string w that belongs to V*. To 
be more precise, 

i. if w — e, then A is popped from the stack, whereas 

ii. if w = BiB 2 ... B k , with k > 1 and B 1 , B 2 , ..., B k E T, then 
A is replaced by w, and B k becomes the new top symbol of 
the stack. 

Later, we will specify when the pushdown automaton accepts the input 
string. 

We now give a formal definition of a deterministic pushdown automaton. 

Definition 3.5.1 A deterministic pushdown automaton is a 5-tuple M = 
(£, T, Q, 5, q ), where 
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1. E is a finite set, called the tape alphabet ; the blank symbol □ is not 
contained in E, 

2 . T is a finite set, called the stack alphabet ; this alphabet contains the 
special symbol $, 

3. Q is a finite set, whose elements are called states, 

4. g is an element of Q: it is called the start state, 

5. 6 is called the transition function, which is a function 

5 : Q x (E U {□}) x T Q x {iV, R] x T*. 

The transition function 5 can be thought of as being the “program” of the 
pushdown automaton. This function tells us what the automaton can do in 
one “computation step”: Let r e Q, a e E U {□}, and deb Furthermore, 
let r' e Q, a e {R, TV}, and w € T* be such that 

5(r, a, A) — (r', a, w). (3.1) 

This transition means that if 

• the pushdown automaton is in state r, 

• the tape head reads the symbol a, and 

• the top symbol on the stack is A, 
then 


• the pushdown automaton switches to state r', 

• the tape head moves according to a: if a = R, then it moves one cell 
to the right; if a = N, then it does not move, and 

• the top symbol A on the stack is replaced by the string w. 

We will write the computation step (3.1) in the form of the instruction 
raA —> r'crw. 

We now specify the computation of the pushdown automaton M = (E, T, Q, S, q). 
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Start configuration: Initially, the pushdown automaton is in the start state 
q, the tape head is on the leftmost symbol of the input string Gqa 2 ... a n , and 
the stack only contains the special symbol $. 

Computation and termination: Starting in the start configuration, the 
pushdown automaton performs a sequence of computation steps as described 
above. It terminates at the moment when the stack becomes empty. (Hence, 
if the stack never gets empty, the pushdown automaton does not terminate.) 

Acceptance: The pushdown automaton accepts the input string aia 2 ... a n £ 
£*, if 

1. the automaton terminates on this input, and 

2. at the time of termination (i.e., at the moment when the stack gets 
empty), the tape head is on the cell immediately to the right of the cell 
containing the symbol a n (this cell must contain the blank symbol □). 

In all other cases, the pushdown automaton rejects the input string. Thus, 
the pushdown automaton rejects this string if 

1. the automaton does not terminate on this input (i.e., the computation 
“loops forever”) or 

2. at the time of termination, the tape head is not on the cell immediately 
to the right of the cell containing the symbol a n . 

We denote by L(M) the language accepted by the pushdown automaton 
M. Thus, 

L(M) = {w € £* : M accepts w}. 

The pushdown automaton described above is deterministic. For a non- 
deterministic pushdown automata, the current computation step may not 
be uniquely defined, but the automaton can make a choice out of a finite 
number of possibilities. In this case, the transition function <5 is a function 

TQx(SU {□}) xT4 V f {Q x {A, R} x T*), 

where Vf(K ) is the set of all finite subsets of the set K. 

We say that a nondeterministic pushdown automaton M accepts an input 
string, if there exists an accepting computation, in the sense as described for 
deterministic pushdown automata. We say that M rejects an input string, if 
every computation on this string is rejecting. As before, we denote by L(M ) 
the set of all strings in £* that are accepted by M. 
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3.6 Examples of pushdown automata 

3.6.1 Properly nested parentheses 

We will show how to construct a deterministic pushdown automaton, that 
accepts the set of all strings of properly nested parentheses. Observe that a 
string w in {(,)}* is properly nested if and only if 


• in every prefix of re, the number of “(” is greater than or equal to the 
number of and 


• in the complete string w, the number of “(” is equal to the number of 


We will use the tape symbol a for “(”, and the tape symbol b for 

The idea is as follows. Recall that initially, the stack only contains the 
special symbol $. The only purpose of this symbol is to enable us to check 
whether or not the stack is empty. The pushdown automaton reads the input 
string from left to right. For every a it reads, it pushes the symbol S onto 
the stack, and for every b it reads, it pops the top symbol from the stack. In 
this way, the number of symbols S on the stack will always be equal to the 
number of as that have been read minus the number of bs that have been 
read; additionally, the bottom of the stack will contain the special symbol $. 
The input string is properly nested if and only if (i) this difference is always 
non-negative and (ii) this difference is zero once the entire input string has 
been read. Hence, the input string is accepted if and only if during this 
process, (i) the stack always contains at least the special symbol $ and (ii) 
at the end, the stack only contains the special symbol $ (which will then be 
popped in the final step). 

Based on this discussion, we obtain the deterministic pushdown automa¬ 
ton M — (E,T,Q,S,q), where £ = {a, b}, F = {$,5}, Q = {g}, and the 
transition function 8 is specified by the following instructions: 
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qa$ qR$S 
qaS ->• qRSS 
qbS —> qRe 

qb$ —l qNe 


#□$ —>■ qNe 
qnS -> qNS 


because of the a, S is pushed onto the stack 
because of the a, S is pushed onto the stack 
because of the b, the top element is popped 
from the stack 

the number of 6s read is larger than the number 
of as read; the stack is made empty (hence, 
the computation terminates before the entire 
string has been read), and the input string is rejected 
the entire input string has been read; the stack is 
made empty, and the input string is accepted 
the entire input string has been read, it contains 
more as than 6s; no changes are made (thus, the 
automaton does not terminate), and the input string 
is rejected 


3.6.2 Strings of the form 0 n l n 

We construct a deterministic pushdown automata that accepts the language 
{0 n l n : n > 0}. 

The automaton uses two states q 0 and q\ , where q 0 is the start state. 
Initially, the automaton is in state q 0 . 

• For each 0 that it reads, the automaton pushes one symbol S onto the 
stack and stays in state g 0 - 

• When the first 1 is read, the automaton switches to state q\. From that 
moment, 

- for each 1 that is read, the automaton pops the top symbol from 
the stack and stays in state q \; 

— if a 0 is read, the automaton does not make any change and, 
therefore, does not terminate. 

Based on this discussion, we obtain the deterministic pushdown automa¬ 
ton M — (£, T, Q, S, q 0 ), where £ = {0,1}, F = {$,5}, Q = Qo is 

the start state, and the transition function 8 is specified by the following 
instructions: 
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go0$ —y qoR$S 
q 0 OS ->■ q 0 RSS 
5ol$ QqN$ 
qo^-S qiRe 
g 0 n$ —y q^Ne 

qo^S —y qoNS 
5i0$ —y q±N$ 
5105" —y q\NS 
5il$ —y q±N$ 
q±\S —y qiRe 

qjO$ —y qiNe 
q\OS — y qiNS 


push S onto the stack 

push S onto the stack 

first symbol in the input is 1; loop forever 

first 1 is encountered 

input string is empty; accept 

input only consists of Os; loop forever 

0 to the right of 1; loop forever 

0 to the right of 1; loop forever 

too many Is; loop forever 

pop top symbol from the stack 

accept 

too many Os; loop forever 


3.6.3 Strings with b in the middle 

We will construct a nondeterministic pushdown automaton that accepts the 
set L of all strings in {a, b}* having an odd length and whose middle symbol 
is b , i.e., 

L = {vbw : v & {a, b}*, w 6 {a, b}*, |u| = |w|}. 

The idea is as follows. The automaton uses two states q and q', where q 
is the start state. These states have the following meaning: 

• If the automaton is in state q, then it has not reached the middle symbol 
of the input string. 

• If the automaton is in state q 1 , then it has read the middle symbol. 

Observe that since the automaton can only make one single pass over the 
input string, it has to “guess” (i.e., use nondeterminism) when it reaches the 
middle of the string. 

• If the automaton is in state q, then, when reading the current tape 
symbol, 

- it pushes one symbol S onto the stack and stays in state q or 

- in case the current tape symbol is b, it “guesses” that it has reached 
the middle of the input string, by switching to state q 1 . 

• If the automaton is in state q', then, when reading the current tape 
symbol, it pops the top symbol S from the stack and stays in state q'. 
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The input string is accepted if and only if the stack is empty for the first 
time, after the entire input string has been read. 

We obtain the nondeterministic pushdown automaton M — ( T,,T,Q,S,q ), 
where E = {a, b}, T = {$, S'}, Q = {q, q'}, q is the start state, and the 
transition function 8 is specified by the following instructions: 


qa$ —> qR$S 
qaS ->■ qRSS 
qb$ —y q' R$ 
qb$ —y qR$S 
qbS -> q'RS 
qbS -> qRSS 
gD$ — > qN$ 
qUS qNS 
q'a$ —> q'Ne 

q'aS —>• q'Re 
q'b% —y q'Ne 

q'bS —> q'Re 
q r O$ —>■ q'Ne 
q'OS -»■ q'NS 


push S onto the stack 
push S onto the stack 
reached the middle 

did not reach the middle; push S onto the stack 
reached the middle 

did not reach the middle; push S onto the stack 
loop forever 
loop forever 

stack is empty; terminate, but reject, because 

the entire input string has not been read 

pop top symbol from stack 

stack is empty; terminate, but reject, because 

the entire input string has not been read 

pop top symbol from stack 

accept 

loop forever 


Remark 3.6.1 It can be shown that there is no deterministic pushdown 
automaton that accepts the language L. The reason is that a deterministic 
pushdown automaton cannot determine when it reaches the middle of the 
input string. Thus, unlike as for finite automata, nondeterministic pushdown 
automata are more powerful than their deterministic counterparts. 


3.7 Equivalence of pushdown automata and 
context-free grammars 

The main result of this section is that nondeterministic pushdown automata 
and context-free grammars are equivalent in power: 

Theorem 3.7.1 Let E be an alphabet and let A C E* be a language. Then 
A is context-free if and only if there exists a nondeterministic pushdown 
automaton that accepts A. 
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We will prove one direction of this theorem. That is, we will show how to 
convert an arbitrary context-free grammar to a nondeterministic pushdown 
automaton. 

Let G = ( V ,, E, 1Z, $) be a context-free grammar, where V is the set of 
variables, E is the set of terminals, 7Z is the set of rules, and $ is the start 
variable. By Theorem 3.4.2, we may assume that G is in Chomsky normal 
form. Hence, every rule in 1Z has one of the following three forms: 

1. A —y BC, where A, B, and C are variables, B ^ $, and C ^ $. 

2 . A —y a, where A is a variable and a is a terminal. 

3. $ — y e. 

We will construct a nondeterministic pushdown automaton M that ac¬ 
cepts the language L(G) of this grammar G. Observe that M must have the 
following property: For every string w = a\Ci 2 ... a n G E*, 

w E L{G) if and only if M accepts w. 

This can be reformulated as follows: 

$ =>■ aia 2 ... a n 

if and only if there exists a computation of M that starts in the initial 
configuration 


M "■ H 


35 ] 


and ends in the configuration 

M N N D I 

I —S 
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where [0] indicates that the stack is empty. 

Assume that $ =>■ aia 2 ... a n . Then there exists a derivation (using the 
rules of 1Z) of the string aia 2 ... a n from the start variable $. We may assume 
that in each step in this derivation, a rule is applied to the leftmost variable in 
the current string. Hence, at any moment during the derivation, the current 
string has the form 

aia 2 ... a,i_iA k A k _i... Ai t (3.2) 

for some integers i and k with 1 < i < n + 1 and k > 0, and variables 
Ai, A 2 , ..., A k . (In particular, at the start of the derivation, we have i — 1 
and k = 1, and the current string is A k = $. At the end of the derivation, 
we have i — n + 1 and k = 0, and the current string is a\a 2 ... a n .) 

We will define the pushdown automaton M in such a way that the current 
string (3.2) corresponds to the configuration 


N ' N ' fit 5 ! 


At 


M 


Based on this discussion, we obtain the nondeterministic pushdown au¬ 
tomaton M = (E, V, {g}, S, g), where 

• the tape alphabet is the set E of terminals of G, 

• the stack alphabet is the set V of variables of G, 

• the set of states consists of one state q, which is the start state, and 

• the transition function 5 is obtained from the rules in 7 Z, in the following 
way: 


- For each rule in 1Z that is of the form A —>■ BC, with A,B,Ce V, 
the pushdown automaton M has the instructions 

qaA —»• qNCB, for all a e E. 
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- For each rule in TZ that is of the form A —>• a, with A E V and 
a E E, the pushdown automaton M has the instruction 

qaA —>• qRe. 

— If TZ contains the rule $ —>• e, then the pushdown automaton M 
has the instruction 

q\3$ qNe. 

This concludes the definition of M. It remains to prove that L(M ) = 
L(G), i.e., the language of the nondeterministic pushdown automaton M is 
equal to the language of the context-free grammar G. Hence, we have to 
show that for every string w E E*, 

w E L(G) if and only if w E L(M), 

which can be rewritten as 

$ => w if and only if M accepts w. 

Claim 3.7.2 Let aia 2 ... a n be a string in £* ; let Ai,A 2 , ..., A k be variables 
in V, and let i and k be integers with 1 < i < n + 1 and k > 0. Then the 
following holds: 

$ a\a 2 ■ ■ ■ ai—\A] t A] t _\... A\ 

if and only if there exists a computation of M from the initial configuration 

M "■ H ■■■ M D I 

I —*[3 

to the configuration 


B i a -i m 


Ak 

A[ 
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Proof. The claim can be proved by induction. Let 
w = aia 2 ■ . . a,i_iA k A k _i.. ,A%. 

Assume that k > 1 and assume that the claim is true for the string w. Then 
we have to show that the claim is still true after applying a rule in 1Z to the 
leftmost variable A k in w. Since the grammar is in Chomsky normal form, 
the rule to be applied is either of the form A k —> BC or of the form A k —»• a*. 
In both cases, the property mentioned in the claim is maintained. ■ 

We now use Claim 3.7.2 to prove that L(M) = L{G). Let w = a k a 2 ... a n 
be an arbitrary string in E*. By applying Claim 3.7.2, with i — n + 1 and 
k — 0, we see that w 6 L(G), i.e., 

S - GL\Oj 2 . . . Cbfl , 

if and only if there exists a computation of M from the initial configuration 

N- - N- - HD 

I —a 

to the configuration 

M ■■■ l a I ■ N D I 

i —s 


But this means that w G L(G) if and only if the automaton M accepts the 
string w. 

This concludes the proof of the fact that every context-free grammar can 
be converted to a nondeterministic pushdown automaton. As mentioned 
already, we will not give the conversion in the other direction. We finish this 
section with the following observation: 

Theorem 3.7.3 Let E be an alphabet and let A C E* be a context-free lan¬ 
guage. Then there exists a nondeterministic pushdown automaton that ac¬ 
cepts A and has only one state. 
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Proof. Since A is context-free, there exists a context-free grammar G 0 such 
that L(G 0 ) = A. By Theorem 3.4.2, there exists a context-free grammar G 
that is in Chomsky normal form and for which L(G) — L{Gq). The construc¬ 
tion given above converts G to a nondeterministic pushdown automaton M 
that has only one state and for which L(M) = L(G). I 


3.8 The pumping lemma for context-free lan¬ 
guages 

In Section 2.9, we proved the pumping lemma for regular languages and 
used it to prove that certain languages are not regular. In this section, we 
generalize the pumping lemma to context-free languages. The idea is to 
consider the parse tree (see Section 3.1) that describes the derivation of a 
sufficiently long string in the context-free language L. Since the number of 
variables in the corresponding context-free grammar G is finite, there is at 
least one variable, say Aj, that occurs more than once on the longest root- 
to-leaf path in the parse tree. The subtree which is sandwiched between two 
occurrences of Aj on this path can be copied any number of times. This will 
result in a legal parse tree and, hence, in a “pumped” string that is in the 
language L. 


Theorem 3.8.1 (Pumping Lemma for Context-Free Languages) Let 

L be a context-free language. Then there exists an integer p > 1, called the 
pumping length, such that the following holds: Every string s in L, with 
|s| > p, can be written as s = uvxyz, such that 

1. \vy\ > 1 (i.e., v and y are not both empty), 

2. \vxy\ < p, and 

3. uv l xy l z G L, for all i> 0. 

3.8.1 Proof of the pumping lemma 

The proof of the pumping lemma will use the following result about parse 
trees: 
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Lemma 3.8.2 Let G be a context-free grammar in Chomsky normal form, 
let s be a non-empty string in L(G), and let T be a parse tree for s. Let Z be 
the height of T, i.e., Z is the number of edges on a longest root-to-leaf path 
in T. Then 

|s| < 2 l ~ x . 

Proof. The claim can be proved by induction on i. By looking at some 
small values of Z and using the fact that G is in Chomsky normal form, you 
should be able to verify the claim. ■ 

Now we can start with the proof of the pumping lemma. Let L be a 
context-free language and let E be the alphabet of L. By Theorem 3.4.2, there 
exists a context-free grammar in Chomsky normal form, G = (V, E ,R,S), 
such that L = L(G). 

Define r to be the number of variables of G and define p = 2 r . We will 
prove that the value of p can be used as the pumping length. Consider an 
arbitrary string s in L such that |s| > p, and let T be a parse tree for s. Let 
Z be the height of T. Then, by Lemma 3.8.2, we have 

M < 2 e ~\ 

On the other hand, we have 

|s| >p — 2 r . 

By combining these inequalities, we see that 2 r < 2f ~ 1 , which can be rewrit¬ 
ten as 

Z>r+ 1. 

Consider the nodes on a longest root-to-leaf path in T. Since this path 
consists of Z edges, it consists of Z + 1 nodes. The first Z of these nodes store 
variables, which we denote by A [h A 1: ..., A(_ x (where A 0 = S), and the last 
node (which is a leaf) stores a terminal, which we denote by a. 

Since Z — 1 — r > 0, the sequence 

A^—l—r, An_ r , ■ ■ ■ , Ai-\ 

of variables is well-defined. Observe that this sequence consists of r + 1 
variables. Since the number of variables in the grammar G is equal to r, 
the pigeon hole principle implies that there is a variable that occurs at least 
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twice in this sequence. In other words, there are indices j and k, such that 
t — 1 — r < j < k < £ — 1 and Aj = A k . Refer to the figure below for an 
illustration. 



Recall that T is a parse tree for the string s. Therefore, the terminals 
stored at the leaves of T, in the order from left to right, form s. As indicated 
in the figure above, the nodes storing the variables Aj and A k partition s 
into five substrings u, v, x, y, and z, such that s — uvxyz. 

It remains to prove that the three properties stated in the pumping lemma 
hold. We start with the third property, i.e., we prove that 

uv l xy l z G L, for all i > 0. 
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In the grammar G, we have 


S => uAjz. 

(3.3) 

Since Aj =4* vA k y and A k = Aj , we have 


Aj =4- vAjy. 

(3.4) 

Finally, since A k x and A k = Aj , we have 


Aj =4> x. 

(3.5) 


From (3.3) and (3.5), it follows that 

S =>■ uAjz =>■ uxz, 

which implies that the string uxz is in the language L. Similarly, it follows 
from (3.3), (3.4), and (3.5) that 

S => uAjZ =>• uvAjyz =>• uvvAjyyz =>- uvvxyyz. 

Hence, the string uv 2 xy 2 z is in the language L. In general, for each i > 0, 
the string uv l xy l z is in the language L, because 

S =>- uAjz =>• uv l Ajy l z =>• uv l xy l z. 

This proves that the third property in the pumping lemma holds. 

Next we show that the second property holds. That is, we prove that 
\vxy\ < p. Consider the subtree rooted at the node storing the variable 
Aj. The path from the node storing Aj to the leaf storing the terminal 
a is a longest path in this subtree. (Convince yourself that this is true.) 
Moreover, this path consists of £ — j edges. Since Aj =>■ vxy, this subtree 
is a parse tree for the string vxy (where Aj is used as the start variable). 
Therefore, by Lemma 3.8.2, we can conclude that \vxy\ < We know 

that l — 1 — r < j, which is equivalent to £ — j — 1 < r. It follows that 

\vxy\ < <2 r = p. 

Finally, we show that the first property in the pumping lemma holds. 
That is, we prove that \vy\ > 1. Recall that 

Aj vA k y. 
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Let the first rule used in this derivation be Aj —> BC. (Since the variables 
Aj and A k , even though they are equal, are stored at different nodes of the 
parse tree, and since the grammar G is in Chomsky normal form, this first 
rule exists.) Then 

Aj =>- BC =>- vA k y. 

Observe that the string BC has length two. Moreover, by applying rules of 
a grammar in Chomsky normal form, strings cannot become shorter. (Here, 
we use the fact that the start variable does not occur on the right-hand side 
of any rule.) Therefore, we have \vA k y\ > 2. But this implies that \vy\ > 1. 
This completes the proof of the pumping lemma. 

3.8.2 Applications of the pumping lemma 

First example 

Consider the language 

A = {a n b n c n : n > 0}. 

We will prove by contradiction that A is not a context-free language. 

Assume that A is a context-free language. Let p > 1 be the pumping 
length, as given by the pumping lemma. Consider the string s = a p b p c p . 
Observe that s 6 A and |s| = 3 p > p. Hence, by the pumping lemma, s can 
be written as s — uvxyz, where \vy\ > 1, \vxy\ < p , and uv' l xy l z e A for all 
i > 0. 

Observe that the pumping lemma does not tell us the location of the 
substring vxy in the string s, it only gives us an upper bound on the length 
of this substring. Therefore, we have to consider three cases, depending on 
the location of vxy in s. 

Case 1: The substring vxy does not contain any c. 

Consider the string uv 2 xy 2 z = uvvxyyz. Since \vy\ > 1, this string 
contains more than p many as or more than p many bs. Since it contains 
exactly p many cs, it follows that this string is not in the language A. This 
is a contradiction because, by the pumping lemma, the string uv 2 xy 2 z is in 
A. 

Case 2: The substring vxy does not contain any a. 

Consider the string uv 2 xy 2 z = uvvxyyz. Since \vy\ > 1, this string 
contains more than p many bs or more than p many cs. Since it contains 
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exactly p many as, it follows that this string is not in the language A. This 
is a contradiction because, by the pumping lemma, the string uv 2 xy 2 z is in 
A. 

Case 3: The substring vxy contains at least one a and at least one c. 

Since s = a p b p c p , this implies that \vxy\ > p, which again contradicts the 
pumping lemma. 

Thus, in all of the three cases, we have obtained a contradiction. There¬ 
fore, we have shown that the language A is not context-free. 

Second example 

Consider the languages 


A = {ww R : w € {a, 6}*}, 

where w R is the string obtained by writing w backwards, and 
B = {ww : w e {a, b}*}. 

Even though these languages look similar, we will show that A is context-free 
and B is not context-free. 

Consider the following context-free grammar, in which S is the start vari¬ 
able: 

S -)• e\aSa\bSb. 

It is easy to see that the language of this grammar is exactly the language A. 
Therefore, A is context-free. Alternatively, we can show that A is context- 
free, by constructing a (nondeterministic) pushdown automaton that accepts 
A. This automaton has two states q and q', where q is the start state. If the 
automaton is in state q, then it did not yet finish reading the leftmost half of 
the input string; it pushes all symbols read onto the stack. If the automaton 
is in state q ', then it is reading the rightmost half of the input string; for each 
symbol read, it checks whether it is equal to the symbol on top of the stack 
and, if so, pops the top symbol from the stack. The pushdown automaton 
uses nondeterminism to “guess” when to switch from state q to state q' (i.e., 
when it has completed reading the leftmost half of the input string). 

At this point, you should convince yourself that the two approaches above, 
which showed that A is context-free, do not work for B. The reason why 
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they do not work is that the language B is not context-free, as we will prove 
now. 

Assume that B is a context-free language. Let p > 1 be the pumping 
length, as given by the pumping lemma. At this point, we must choose a 
string s in B, whose length is at least p, and that does not satisfy the three 
properties stated in the pumping lemma. Let us try the string s = a p ba p b. 
Then s e B and \s\ = 2p + 2 > p. Hence, by the pumping lemma, s can be 
written as s = uvxyz , where (i) \vy\ > 1, (ii) \vxy\ < p, and (iii) uv l xy l z e B 
for all i > 0. It may happen that p > 3, u = a p_1 , v = a, x = b, y = a, 
and z = a p ~ x b. If this is the case, then properties (i), (ii), and (iii) hold, 
and, thus, we do not get a contradiction. In other words, we have chosen 
the “wrong” string s. This string is “wrong”, because there is only one b 
between the as. Because of this, v can be in the leftmost block of as, and 
y can be in the rightmost block of as. Observe that if there were at least p 
many bs between the as, then this would not happen, because \vxy\ < p. 

Based on the discussion above, we choose s = aPlfaPlf. Observe that 
s G B and \s\ — Ap>p. Hence, by the pumping lemma, s can be written as 
s — uvxyz, where \vy\ > 1, \vxy\ < p, and uv l xy l z 6 B for all i > 0. Based 
on the location of vxy in the string s, we distinguish three cases: 

Case 1: The substring vxy overlaps both the leftmost half and the rightmost 
half of s. 

Since \vxy\ < p, the substring vxy is contained in the “middle” part of s, 
i.e., vxy is contained in the block ffa p . Consider the string uv°xy°z = uxz. 
Since \vy\ > 1, we know that at least one of v and y is non-empty. 

• If v ^ e, then v contains at least one b from the leftmost block of bs in 
s , whereas y does not contain any b from the rightmost block of bs in s. 
Therefore, in the string uxz , the leftmost block of bs contains fewer bs 
than the rightmost block of bs. Hence, the string uxz is not contained 
in B. 

• If y e, then y contains at least one a from the rightmost block of 
as in s, whereas v does not contain any a from the leftmost block of 
as in s. Therefore, in the string uxz , the leftmost block of as contains 
more as than the rightmost block of as. Hence, the string uxz is not 
contained in B. 

In both cases, we conclude that the string uxz is not an element of the 
language B. But, by the pumping lemma, this string is contained in B. 
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Case 2: The substring vxy is in the leftmost half of s. 

In this case, none of the strings uxz , uv 2 xy 2 z, uv 3 xy 3 z, uv 4 xy 4 z, etc., 
is contained in B. But, by the pumping lemma, each of these strings is 
contained in B. 

Case 3: The substring vxy is in the rightmost half of s. 

This case is symmetric to Case 2: None of the strings uxz, uv 2 xy 2 z, 
uv 3 xy 3 z , uv 4 xy 4 z, etc., is contained in B. But, by the pumping lemma, each 
of these strings is contained in B. 

To summarize, in each of the three cases, we have obtained a contradic¬ 
tion. Therefore, the language B is not context-free. 

Third example 

We have seen in Section 3.2.4 that the language 

{a m b n c m+n : m > 0, n > 0} 

is context-free. Using the pumping lemma for regular languages, it is easy to 
prove that this language is not regular. In other words, context-free gram¬ 
mars can verify addition, whereas finite automata are not powerful enough 
for this. We now consider the problem of verifying multiplication: Let A be 
the language defined as 


A = {a m b n c mn : m > 0, n > 0}. 

We will prove by contradiction that A is not a context-free language. 

Assume that A is context-free. Let p > 1 be the pumping length, as 
given by the pumping lemma. Consider the string s = a p b p c p . Then, s <G A 
and |s| = 2 p + p 2 > p. Hence, by the pumping lemma, s can be written as 
s — uvxyz , where \vy\ > 1, \vxy\ < p, and uv' l xy l z e A for all i > 0. 

There are three possible cases, depending on the location of vxy in the 
string s. 

Case 1: The substring vxy is completely contained in the block of cs in s. 

Consider the string uv 2 xy 2 z. Since \vy\ > 1, the string uv 2 xy 2 z consists 
of p many as, p many bs, but more than p 2 many cs. Therefore, this string 
is not contained in A. But, by the pumping lemma, it is contained in A. 

Case 2: The substring vxy does not contain any c. 
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Consider again the string uv 2 xy 2 z. This string consists of p 2 many cs. 
Since \vy\ > 1, in the string uv 2 xy 2 z , the number of as multiplied by the 
number of bs is larger than p 2 . Therefore, uv 2 xy 2 z is not contained in A. 
But, by the pumping lemma, this string is contained in A. 

Case 3: The substring vxy contains at least one b, at least one c, and no a. 

In this case, we can write vy = Vc k , where j > 0, k > 0, and j + k > 1. 
Consider the string uxz. We can write this string as uxz — a p b p ~ j c p ~ k . 
Since, by the pumping lemma, this string is contained in A, we have p(p—j) = 
p 2 — k, which implies that jp — k. 

If j = 0, then k = 0, which contradicts the fact that j + k > 1. Therefore, 
j > 1. It follows that k = jp>p and 

\vxy\ > \vy\ = j + k> 1 + p. 

But, by the pumping lemma, we have \vxy\ < p. 

Observe that, since \vxy\ < p, the above three cases cover all possibilities 
for the location of vxy in the string s. In each of the three cases, we have 
obtained a contradiction. Therefore, the language A is not context-free. 


Exercises 

3.1 Construct context-free grammars that generate the following languages. 
In all cases, E = {0,1}. 

• {0 2n l" : n > 0} 

• {w : w contains at least three Is} 

• {w : the length of w is odd and its middle symbol is 0} 

• {w : w is a palindrome}. 

A palindrome is a string w having the property that w = w R , i.e., 
reading w from left to right gives the same result as reading w from 
right to left. 

• {w : w starts and ends with the same symbol} 

• {w : w starts and ends with different symbols} 
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3.2 Let G = (V, E, i?, S') be the context-free grammar, where V = {^4, B , S'}, 
E = {0,1}, S'is the start variable, and R consists of the rules 

S ->• 0S'|L4|e 
A -)■ 05|15 
5 ->• QA\1B 

Define the following language L: 

L = {w G (0,1}* : w is the binary representation of a non-negative 
integer that is divisible by three } U {e}. 

Prove that L = L(G). (Hint: The variables S', A, and B are used to 
remember the remainder after division by three.) 

3.3 Let G = (V, E, R, S') be the context-free grammar, where V = {^4, B, S'}, 
E = {a, b}, S is the start variable, and R consists of the rules 

S aB\bA 

A a\aS\BAA 

B b\bS\ABB 

• Prove that ababba 6 L(G). 

• Prove that L(G) is the set of all non-empty strings w over the alphabet 
{a, b} such that the number of as in w is equal to the number of bs in 
w. 


3.4 Let A and B be context-free languages over the same alphabet E. 

• Prove that the union A U B of A and B is also context-free. 

• Prove that the concatenation AB of A and B is also context-free. 

• Prove that the star A* of A is also context-free. 

3.5 Define the following two languages A and B: 

A = {a m b n c n : m > 0, n > 0} 

and 

B = {a m b rn c n : m > 0, n > 0}. 
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• Prove that both A and B are context-free, by constructing two context- 
free grammars, one that generates A and one that generates B. 

• We have seen in Section 3.8.2 that the language 

{a n b n c n : n > 0} 

is not context-free. Explain why this implies that the intersection of 
two context-free languages is not necessarily context-free. 

• Use De Morgan’s Law to conclude that the complement of a context- 
free language is not necessarily context-free. 

3.6 Let A be a context-free language and let B be a regular language. 

• Prove that the intersection A D B of A and B is context-free. 

• Prove that the set-difference 

A \ B = {w : w 6 A, w B} 
of A and B is context-free. 

• Is the set-difference of two context-free languages necessarily context- 
free? 

3.7 Let L be the language consisting of all non-empty strings w over the 
alphabet {a, b} such that 

• the number of as in w is equal to the number of bs in w, 

• w does not contain the substring abba , and 

• w does not contain the substring bbaa. 

In this exercise, you will prove that L is context-free. 

Let A be the language consisting of all non-empty strings w over the 
alphabet {a, b} such that the number of as in w is equal to the number of bs 
in w. In Exercise 3.3, you have shown that A is context-free. 

Let B be the language consisting of all strings w over the alphabet {a, b} 
such that 
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• w does not contain the substring abba, and 

• w does not contain the substring bbaa. 

1. Give a regular expression that describes the complement of B. 

2. Argue that B is a regular language. 

3. Use Exercise 3.6 to argue that L is a context-free language. 

3.8 Construct (deterministic or nondeterministic) pushdown automata that 
accept the following languages. 

1. {0 2n l n : n > 0}. 

2. {0 n l m 0 n : n > l,m> 1}. 

3. {w G {0,1}* : w contains more Is than Os}. 

4. {ww R : w G (0,1}*}. 

(If w — Wi... w n , then w R = w n ... w^.) 

5. {w e {0,1}* : w is a palindrome}. 

3.9 Let L be the language 

L = {a m b n : 0 <m<n< 2m}. 

1. Prove that L is context-free, by constructing a context-free grammar 
whose language is equal to L. 

2. Prove that L is context-free, by constructing a nondeterministic push¬ 
down automaton that accepts L. 

3.10 Prove that the following languages are not context-free. 

• {a n ba 2n ba 3n :n> 0}. 

• {a n b n a n b n : n > 0}. 

• {a m b n c k : m > 0, n > 0, k = max(m, n)}. 
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• {w#x : w is a substring of x , and w, x 6 {a, &}*}. 

For example, the string aba#abbababbb is in the language, whereas the 
string aba^baabbaabb is not in the language. The alphabet is {a, b, #}. 


{ w e {a, b, c}* : w contains more 6’s than a’s and 
w contains more c’s than a’s }. 


• {1” : n is a prime number}. 

• {( ab n ) n : n > ()}. (The parentheses are not part of the alphabet; thus, 
the alphabet is {a, b , }.) 

3.11 Let L be a language consisting of finitely many strings. Show that L 
is regular and, therefore, context-free. Let k be the maximum length of any 
string in L. 

• Prove that every context-free grammar in Chomsky normal form that 
generates L has more than log k variables. (The logarithm is in base 
2-) 

• Prove that there is a context-free grammar that generates L and that 
has only one variable. 

3.12 Let L be a context-free language. Prove that there exists an integer 
p > 1, such that the following is true: For every string s in L with s > p, 
there exists a string s' in L such that |.s| < |s'| < |s| + p. 
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Chapter 4 


Turing Machines and the 
Church-Turing Thesis 


In the previous chapters, we have seen several computational devices that 
can be used to accept or generate regular and context-free languages. Even 
though these two classes of languages are fairly large, we have seen in Sec¬ 
tion 3.8.2 that these devices are not powerful enough to accept simple lan¬ 
guages such as A = {a m b n c mn : m > 0, n > 0}. In this chapter, we introduce 
the Turing machine, which is a simple model of a real computer. Turing ma¬ 
chines can be used to accept all context-free languages, but also languages 
such as A. We will argue that every problem that can be solved on a real 
computer can also be solved by a Turing machine (this statement is known 
as the Church-Turing Thesis). In Chapter 5, we will consider the limitations 
of Turing machines and, hence, of real computers. 


4.1 Definition of a Turing machine 

We start with an informal description of a Turing machine. Such a machine 
consists of the following, see also Figure 4.1. 

1. There are k tapes, for some fixed k > 1. Each tape is divided into 
cells, and is infinite both to the left and to the right. Each cell stores 
a symbol belonging to a finite set T, which is called the tape alphabet. 
The tape alphabet contains the blank symbol □. If a cell contains □, 
then this means that the cell is actually empty. 
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state control 

n 



|l=l|l=l|l=l|a|a|5|a|6|6|a|6|a 

6 |0|Q|0| 


1 


• • (□(□Ini & |a|a.| 6 Inlnlnl 


Figure 4.1: A Turing machine with k = 2 tapes. 


2. Each tape has a tape head which can move along the tape, one cell 
per move. It can also read the cell it currently scans and replace the 
symbol in this cell by another symbol. 

3. There is a state control , which can be in any one of a finite number of 
states. The finite set of states is denoted by Q. The set Q contains 
three special states: a start state , an accept state , and a reject state. 

The Turing machine performs a sequence of computation steps. In one 
such step, it does the following: 

1. Immediately before the computation step, the Turing machine is in a 
state r of Q, and each of the k tape heads is on a certain cell. 

2. Depending on the current state r and the k symbols that are read by 
the tape heads, 

(a) the Turing machine switches to a state r' of Q (which may be 
equal to r), 

(b) each tape head writes a symbol of T in the cell it is currently 
scanning (this symbol may be equal to the symbol currently stored 
in the cell), and 
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(c) each tape head either moves one cell to the left, moves one cell to 
the right, or stays at the current cell. 

We now give a formal definition of a deterministic Turing machine. 

Definition 4.1.1 A deterministic Turing machine is a 7-tuple 

M — (E, r, Qi 8, Qi Qaccepti Qr eject) j 

where 

1. E is a finite set, called the input alphabet ; the blank symbol □ is not 
contained in E, 

2. T is a finite set, called the tape alphabet ; this alphabet contains the 
blank symbol □, and ECT, 

3. Q is a finite set, whose elements are called states, 

4. q is an element of Q] it is called the start state, 

5. q a ccept is an element of Q\ it is called the accept state, 

6- Qr eject is an element of Q; it is called the reject state, 

7. 6 is called the transition function, which is a function 

S : Q x T fc -)■ Q x T fc x {L, R, N} k . 

The transition function 8 is basically the “program” of the Turing ma¬ 
chine. This function tells us what the machine can do in “one computation 
step”: Let r G Q, and let a\, a 2 , ..., a*, e T. Furthermore, let r' G Q, 
a' 1; a' 2 ,..., a' k G T, and a\, a 2 , ..., cr*, G { L, R, N} be such that 

8(r, a i, a 2 , ■ • •, a k ) = (r 7 , a^, a' 2 ,..., a! k , o\, o 2 ,..., a k ). (4.1) 

This transition means that if 

• the Turing machine is in state r, and 

• the head of the Ath tape reads the symbol a*, 1 < i < k, 
then 
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• the Turing machine switches to state r', 

• the head of the i-th tape replaces the scanned symbol a t by the symbol 

1 < i < k, and 

• the head of the i-th tape moves according to cq, 1 < i < k: if cq = L, 
then the tape head moves one cell to the left; if <7j = R , then it moves 
one cell to the right; if a t = N , then the tape head does not move. 

We will write the computation step (4.1) in the form of the instruction 

raia 2 ■ ■ ■ a*, —y r'a^a^ ... a' k aia 2 ... cr fc . 

We now specify the computation of the Turing machine 


M — (E, r, Q , 5 , Qaccepti dreject)- 

Start configuration: The input is a string over the input alphabet E. 
Initially, this input string is stored on the first tape, and the head of this 
tape is on the leftmost symbol of the input string. Initially, all other k — 1 
tapes are empty, i.e., only contain blank symbols, and the Turing machine is 
in the start state q. 

Computation and termination: Starting in the start configuration, the 
Turing machine performs a sequence of computation steps as described above. 
The computation terminates at the moment when the Turing machine en¬ 
ters the accept state q aC ce P t or the reject state q re j ec t■ (Hence, if the Turing 
machine never enters the states q aC ce P t and q r eject, the computation does not 
terminate.) 

Acceptance: The Turing machine M accepts the input string w E E*, if the 
computation on this input terminates in the state q aC ce P t■ If the computation 
on this input terminates in the state q re ject, then M rejects the input string 

w. 

We denote by L(M) the language accepted by the Turing machine M. 
Thus, L(M ) is the set of all strings in E* that are accepted by M. 

Observe that a string w G E* does not belong to L(M) if and only if on 
input w, 

• the computation of M terminates in the state q re ject or 

• the computation of M does not terminate. 


Source: www.csitsansar.com 



4.2. Examples of Turing machines 


137 


4.2 Examples of Turing machines 

4.2.1 Accepting palindromes using one tape 

We will show how to construct a Turing machine with one tape, that decides 
whether or not any input string w G {a, b}* is a palindrome. Recall that the 
string w is called a palindrome, if reading w from left to right gives the same 
result as reading w from right to left. Examples of palindromes are abba, 
baabbbbaab, and the empty string e. 

Start of the computation: The tape contains the input string w, the tape 
head is on the leftmost symbol of w, and the Turing machine is in the start 
state go- 

Idea: The tape head reads the leftmost symbol of w, deletes this symbol 
and “remembers” it by means of a state. Then the tape head moves to 
the rightmost symbol and tests whether it is equal to the (already deleted) 
leftmost symbol. 

• If they are equal, then the rightmost symbol is deleted, the tape head 
moves to the new leftmost symbol, and the whole process is repeated. 

• If they are not equal, the Turing machine enters the reject state, and 
the computation terminates. 

The Turing machine enters the accept state as soon as the string currently 
stored on the tape is empty. 

We will use the input alphabet E = {a, b} and the tape alphabet T = 

{a, b, □}. The set Q of states consists of the following eight states: 

g 0 : start state; tape head is on the leftmost symbol 

q a : leftmost symbol was a; tape head is moving to the right 

q b : leftmost symbol was 6; tape head is moving to the right 

q' a : reached rightmost symbol; test whether it is equal to a, and delete it 

q' b : reached rightmost symbol; test whether it is equal to b, and delete it 

qL : test was positive; tape head is moving to the left 

Qaccept ■ accept state 

q r eject. ’■ reject state 
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The transition function 6 is specified by the following instructions: 


q 0 a ->■ q a OR 
q 0 b ->■ q b OR 
g o n —)■ qaccept 


Q a a -)• q a aR 
q a b ->• q a bR 
q a ° -> q'<PL 


q b a ->• 
q b b ->• g b 6.R 
“4- g£OL 


g„a ->• gz,DL 

q' a b —> q r eject 
^ qaccept 


q b ^ ^ qreject 

q b b —> q^L 

q b C ^ qaccept 


q L a ->■ q L aL 
qi,b qLbL 
qL u —> qo^R 


You should go through the computation of this Turing machine for some 
sample inputs, for example abba , b , abb and the empty string (which is a 
palindrome). 


4.2.2 Accepting palindromes using two tapes 

We again consider the palindrome problem, but now we use a Turing machine 
with two tapes. 

Start of the computation: The first tape contains the input string w and 
the head of the first tape is on the leftmost symbol of w. The second tape is 
empty and its tape head is at an arbitrary position. The Turing machine is 
in the start state g 0 . 

Idea: First, the input string w is copied to the second tape. Then the head 
of the first tape moves back to the leftmost symbol of w, while the head of 
the second tape stays at the rightmost symbol of w. Finally, the actual test 
starts: The head of the first tape moves to the right and, at the same time, 
the head of the second tape moves to the left. While moving, the Turing 
machine tests whether the two tape heads read the same symbol in each 
step. 

The input alphabet is E = {a, b} and the tape alphabet is T = {a, b , □}. 
The set Q of states consists of the following five states: 

g 0 : start state; copy w to the second tape 

gi : w has been copied; head of first tape moves to the left 

g 2 : head of first tape moves to the right; head of second tape moves 

to the left; until now, all tests were positive 
qaccept ■ accept state 
qreject ■ reject State 
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The transition function 6 is specified by the following instructions: 


qo oD —y qoaaRR 
q^bD —y q 0 bbRR 
— y qiOOLL 


qiaa —y q±aaLN 
qiab —y qiabLN 
qiba —y qibaLN 
qibb —qibbLN 
qi Da —>■ q 2 OaRN 
q x Ub —y q 2 DbRN 

{j^OG y Qaccept 


q 2 aa —y q 2 aaRL 
q 2 ab y q r eject 
q 2 bd y q r eject 
q 2 bb —y q 2 bbRL 
q 2 00 y q acc ept 

Again, you should run this Turing machine for some sample inputs. 

4.2.3 Accepting a n b n c n using one tape 

We will construct a Turing machine with one tape that accepts the language 
{a n b n c n : n > 0}. 

Recall that we have proved in Section 3.8.2 that this language is not context- 
free. 

Start of the computation: The tape contains the input string w and the 
tape head is on the leftmost symbol of w. The Turing machine is in the start 
state. 

Idea: In the previous examples, the tape alphabet T was equal to the union 
of the input alphabet E and {□}. In this example, we will add one symbol 
d to the tape alphabet. As we will see, this simplifies the construction of the 
Turing machine. 

The general approach is to repeat the following: Walk along the string 
from left to right, replace the leftmost a by d, replace the leftmost b by d, 
replace the leftmost c by d, and walk back to the leftmost symbol. 
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The input alphabet is E = {a, b, c} and the tape alphabet is Y = {a, b , c, d, □}. 

We use the following states: 

q a : start state; search for the leftmost a 

qb : leftmost a has been replaced by d; 

search for the leftmost b 
q c : leftmost a has been replaced by d; 

leftmost b has been replaced by d; 
search for the leftmost c 
qL : leftmost a has been replaced by d; 

leftmost b has been replaced by d; 
leftmost c has been replaced by d; 
walk to the leftmost symbol 
Qaccept ■ accept state 
Qreject, '■ reject state 

The transition function 8 is specified by the following instructions: 


q a a — y qbdR 

qbCt —> qbCtR 

Qab —> q r eject 

qbb —> q c dR 

Qa/' ^ qr eject 

QbC y Qreject 

q a d —>• q a dR 

qbd — y qbdR 

Qa^ ^ Qaccept 

QlP y Qreject 

q c Ct y qreject 

q L a -y q L aL 

q c b —y q c bR 

qL,b -y- qLbL 

Q c c q L dL 

QlC -> q L cL 

q c d — > q c dR 

qLd -y qLdL 

QcP Qreject 

QlO ->• qaPR 


4.2.4 Accepting a n b n c n using tape alphabet {a, 6, c, □} 

We consider again the language {a n b n c n : n > 0}. In the previous section, 
we presented a Turing machine that uses an extra symbol d. The reader may 
wonder if we can construct a Turing machine for this language that does not 
use any extra symbols. We will show below that this is indeed possible. 

Start of the computation: The tape contains the input string w and the 
tape head is on the leftmost symbol of w. The Turing machine is in the start 
state q 0 . 
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Idea: Repeat the following Stages 1 and 2, until the string is empty. 

Stage 1. Walk along the string from left to right, delete the leftmost a, 
delete the leftmost b, and delete the rightmost c. 

Stage 2. Shift the substring of bs and cs one position to the left; then walk 
back to the leftmost symbol. 

The input alphabet is E = {a, b, c} and the tape alphabet is T = {a, b, c, □}. 
For Stage 1, we use the following states: 

q 0 : start state; tape head is on the leftmost symbol 

q a : leftmost a has been deleted; have not read b 

q b : leftmost b has been deleted; have not read c 

q c : leftmost c has been read; tape head moves to the right 

q' c : tape head is on the rightmost c 

qi : rightmost c has been deleted; tape head is on the rightmost 

symbol or □ 

Qaccept '■ accept state 
q r eject ■ reject state 

The transitions for Stage 1 are specified by the following instructions: 


q^tx —y qaPR 

q a a —>• q a aR 

Qob y qreject 

q a b -y qb^R 

?t)C y qr eject 

QaX- y qr eject 

y Qaccept 

Qa ^ y qr eject 

qba y qr eject 

q c ct y q r eject 

q b b q b bR 

Qc,b y qreject 

q b c -y q c cR 

q c c —y q c cR 

qb FI y qr eject 

q c U —y q' c UL 

we use the following states: 

q' c c —y qjOL 


q\ : as above; tape head is on the rightmost symbol or on □ 
q c : copy c one cell to the left 
q b : copy b one cell to the left 
q 2 : done with shifting; head moves to the left 


Additionally, we use a state q[ which has the following meaning: If the input 
string is of the form a l bc, for some i > 1, then after Stage 1, the tape contains 
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the string a* _1 OQ, the tape head is on the □ immediately to the right of the 
as, and the Turing machine is in state q±. In this case, we move one cell to 
the left; if we then read □, then i — 1, and we accept; otherwise, we read a, 
and we reject. 

The transitions for Stage 2 are specified by the following instructions: 


qia —y cannot happen 

qib —> q r eject 

q\C —y q c OL 
n —y q^oL 

q c a —y cannot happen 
q c b — q b cL 
q c c —y q c cL 

Q O y Qr eject 


q'l a —y q r eject 

q[b —y cannot happen 

q[c —y cannot happen 

y Qaccept 

q b a —>• cannot happen 
q b b -y q b bL 
q b c — y cannot happen 
q b O — y q2bL 


q2d —y q2^L 

q 2 b —y cannot happen 

52e —y cannot happen 

52 □ -y qo n R 

4.2.5 Accepting a m b n c mn using one tape 

We will sketch how to construct a Turing machine with one tape that accepts 
the language 

{d m b n c mn : m > 0, n > 0}. 

Recall that we have proved in Section 3.8.2 that this language is not context- 
free. 

The input alphabet is E = {a, b, c} and the tape alphabet is Y = {a, b, c, $, □}, 
where the purpose of the symbol $ will become clear below. 

Start of the computation: The tape contains the input string w and the 
tape head is on the leftmost symbol of w. The Turing machine is in the start 
state. 

Idea: Observe that a string d m b n c k is in the language if and only if for every 
a, the string contains n many cs. Based on this, the computation consists of 
the following stages: 
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Stage 1. Walk along the input string w from left to right and check whether 
w is an element of the language described by the regular expression a*b*c*. 
If this is not the case, then reject the input string. Otherwise, go to Stage 2. 

Stage 2. Walk back to the leftmost symbol of w. Go to Stage 3. 

Stage 3. In this stage, the Turing machine does the following: 

• Replace the leftmost a by the blank symbol □. 

• Walk to the leftmost b. 

• Zigzag between the bs and cs; each time, replace the leftmost b by the 
symbol $, and replace the rightmost c by the blank symbol □. If, for 
some b, there is no c left, the Turing machine rejects the input string. 

• Continue zigzagging until there are no bs left. Then go to Stage 4. 

Observe that in this third stage, the string a m b n c k is transformed to the 
string a m ~ 1 % n c k ~ n . 

Stage 4. In this stage, the Turing machine does the following: 

• Replace each $ by b. 

• Walk to the leftmost a. 

Hence, in this fourth stage, the string a rn l $ n c k ^ n is transformed to the string 

a m-l Jfi c k-n 

Observe that the input string a m b n c k is in the language if and only if the 
string a m ~ 1 b n c k ~ n is in the language. Therefore, the Turing machine repeats 
Stages 3 and 4, until there are no as left. At that moment, it checks whether 
there are any cs left; if so, it rejects the input string; otherwise, it accepts 
the input string. 

We hope that you believe that this description of the algorithm can be 
turned into a formal description of a Turing machine. 


4.3 Multi-tape Turing machines 

In Section 4.2, we have seen two Turing machines that accept palindromes; 
the first Turing machine has one tape, whereas the second one has two tapes. 
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You will have noticed that the two-tape Turing machine was easier to obtain 
than the one-tape Turing machine. This leads to the question whether multi¬ 
tape Turing machines are more powerful than their one-tape counterparts. 
The answer is “no”: 

Theorem 4.3.1 Let k > 1 be an integer. Any k-tape Turing machine can 
be converted to an equivalent one-tape Turing machine. 

Proof. 1 We will sketch the proof for the case when k — 2. Let M = 
(E, T, Q, 5, q, q a ccept, Qreject ) be a two-tape Turing machine. Our goal is to 
convert M to an equivalent one-tape Turing machine N. That is, N should 
have the property that for all strings iweE*, 

• M accepts w if and only if N accepts w, 

• M rejects w if and only if N rejects w, 

• M does not terminate on input w if and only if N does not terminate 
on input w. 

The tape alphabet of the one-tape Turing machine N is 

r u {± : X e r} u {#}. 

In words, we take the tape alphabet T of M, and add, for each x G L, the 
symbol x. Moreover, we add a special symbol #. 

The Turing machine N will be defined in such a way that any configura¬ 
tion of the two-tape Turing machine M, for example 


• MdolohM - 


I □ I a I a I i. I a !"B] 


1 Thanks to Sergio Cabello for pointing out an error in a previous version of this proof. 
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corresponds to the following configuration of the one-tape Turing machine 
N: 


□ #10 6 l # ° a H a # D 


Thus, the contents of the two tapes of M are encoded on the single tape of 
N. The dotted symbols are used to indicate the positions of the two tape 
heads of M, whereas the three occurrences of the special symbol # are used 
to mark the boundaries of the strings on the two tapes of M. 

The Turing machine N simulates one computation step of M, in the 
following way: 

• Throughout the simulation of this step, N “remembers” the current 
state of M. 

• At the start of the simulation, the tape head of N is on the leftmost 
symbol #. 

• N walks along the string to the right until it finds the first dotted 
symbol. (This symbol indicates the location of the head on the first tape 
of M.) N remembers this first dotted symbol and continues walking 
to the right until it finds the second dotted symbol. (This symbol 
indicates the location of the head on the second tape of M.) Again, N 
remembers this second dotted symbol. 

• At this moment, N is still at the second dotted symbol. N updates 
this part of the tape, by making the change that M would make on its 
second tape. (This change is given by the transition function of M; it 
depends on the current state of M and the two symbols that M reads 
on its two tapes.) 

• N walks to the left until it finds the first dotted symbol. Then, it 
updates this part of the tape, by making the change that M would 
make on its first tape. 

• In the previous two steps, in which the tape is updated, it may be 
necessary to shift a part of the tape. 
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• Finally, N remembers the new state of M and walks back to the left¬ 
most symbol ff. 

It should be clear that the Turing machine N can be constructed by 
introducing appropriate states. ■ 


4.4 The Church-Turing Thesis 

We all have some intuitive notion of what an algorithm is. This notion will 
probably be something like “an algorithm is a procedure consisting of com¬ 
putation steps that can be specified in a finite amount of text”. For example, 
any “computational process” that can be specified by a Java program, should 
be considered an algorithm. Similarly, a Turing machine specifies a “com¬ 
putational process” and, therefore, should be considered an algorithm. This 
leads to the question of whether it is possible to give a mathematical defini¬ 
tion of an “algorithm”. We just saw that every Java program represents an 
algorithm and that every Turing machine also represents an algorithm. Are 
these two notions of an algorithm equivalent? The a n swer is “yes”. In fact, 
the following theorem states that many different notions of “computational 
process” are equivalent. (We hope that you have gained sufficient intuition, 
so that none of the claims in this theorem comes as a surprise to you.) 

Theorem 4.4.1 The following computation models are equivalent, i.e., any 
one of them can be converted to any other one: 

1. One-tape Turing machines. 

2. k-tape Turing machines, for any k > 1. 

3. Non-deterministic Turing machines. 

4■ Java programs. 

5. C++ programs. 

6. Lisp programs. 


Source: www.csitsansar.com 



Exercises 


147 


In other words, if we define the notion of an algorithm using any of the 
models in this theorem, then it does not matter which model we take: All 
these models give the same notion of an algorithm. 

The problem of defining the notion of an algorithm goes back to David 
Hilbert. On August 8, 1900, at the Second International Congress of Math¬ 
ematicians in Paris, Hilbert presented a list of problems that he considered 
crucial for the further development of mathematics. Hilbert’s 10th problem 
is the following: 

Does there exist a finite process that decides whether or not any 
given polynomial with integer coefficients has integral roots? 

Of course, in our language, Hilbert asked whether or not there exists an 
algorithm that decides, when given an arbitrary polynomial equation (with 
integer coefficients) such as 

12x 3 y 7 z 5 + 7x 2 y 4 z - x 4 + y 2 z 7 - z 3 + 10 = 0, 

whether or not this equation has a solution in integers. In 1970, Matiyasevich 
proved that such an algorithm does not exist. Of course, in order to prove 
this claim, we first have to agree on what an algorithm is. In the beginning 
of the twentieth century, mathematicians gave several definitions, such as 
Turing machines (1936) and the A-calculus (1936), and they proved that all 
these are equivalent. Later, after programming languages were invented, it 
was shown that these older notions of an algorithm are equivalent to notions 
of an algorithm that are based on C programs, Java programs, Lisp programs, 
Pascal programs, etc. 

In other words, all attempts to give a rigorous definition of the notion of 
an algorithm led to the same concept. Because of this, computer scientists 
nowadays agree on what is called the Church-Turing Thesis: 


Church-Turing Thesis: Every computational process that is intuitively 
considered to be an algorithm can be converted to a Turing machine. 

In other words, this basically states that we define an algorithm to be a 
Turing machine. At this point, you should ask yourself, whether the Church- 
Turing Thesis can be proved. Alternatively, what has to be done in order to 
disprove this thesis? 


Source: www.csitsansar.com 



148 Chapter 4. Turing Machines and the Church-Turing Thesis 


Exercises 

4.1 Construct a Turing machine with one tape, that accepts the language 

{0 2n l n : n > 0}. 

Assume that, at the start of the computation, the tape head is on the leftmost 
symbol of the input string. 

4.2 Construct a Turing machine with one tape, that accepts the language 

{w : w contains twice as many Os as Is}. 

Assume that, at the start of the computation, the tape head is on the leftmost 
symbol of the input string. 

4.3 Let A be the language 

A = { w e {a,b,c}* : w contains more bs than as and 
w contains more cs than as }. 

Give an informal description (in plain English) of a Turing machine with one 
tape, that accepts the language A. 

4.4 Construct a Turing machine with one tape, that receives as input a non¬ 
negative integer x , and that returns as output the integer x + 1. Integers are 
represented as binary strings. 

Start of the computation: The tape contains the binary representation 
of the input x. The tape head is on the leftmost symbol and the Turing 
machine is in the start state go- For example, if x = 431, the tape looks as 
follows: 


l D l D l D li|i|o|i|Q|i|i|i|il D l D l D l 


End of the computation: The tape contains the binary representation of 
the integer x + 1. The tape head is on the leftmost symbol and the Turing 
machine is in the final state q\. For our example, the tape looks as follows: 
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The Turing machine in this exercise does not have an accept state or a 
reject state; instead, it has a final state gi. As soon as state q\ is entered, 
the Turing machine terminates. At termination, the contents of the tape is 
the output of the Turing machine. 

4.5 Construct a Turing machine with two tapes, that receives as input two 
non-negative integers x and y, and that returns as output the integer x + y. 
Integers are represented as binary strings. 

Start of the computation: The first tape contains the binary represen¬ 
tation of x and its head is on the rightmost symbol of x. The second tape 
contains the binary representation of y and its head is on the rightmost bit 
of y. At the start, the Turing machine is in the start state go- 

End of the computation: The first tape contains the binary representation 
of x and its head is on the rightmost symbol of x. The second tape contains 
the binary representation of the integer x + y (thus, the integer y is “gone”). 
The head of the second tape is on the rightmost bit of x + y. The Turing 
machine is in the final state gi. 

4.6 Give an informal description (in plain English) of a Turing machine with 
one tape, that receives as input two non-negative integers x and y, and that 
returns as output the integer x+y. Integers are represented as binary strings. 
If you are an adventurous student, you may give a formal definition of your 
Turing machine. 

4.7 Construct a Turing machine with one tape, that receives as input an 
integer x > 1, and that returns as output the integer x — 1. Integers are 
represented in binary. 

Start of the computation: The tape contains the binary representation of 
the input x. The tape head is on the rightmost symbol of x and the Turing 
machine is in the start state g 0 . 

End of the computation: The tape contains the binary representation of 
the integer x — 1. The tape head is on the rightmost bit of x — 1 and the 
Turing machine is in the final state gi. 
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4.8 Give an informal description (in plain English) of a Turing machine with 
three tapes, that receives as input two non-negative integers x and y, and 
that returns as output the integer xy. Integers are represented as binary 
strings. 

Start of the computation: The first tape contains the binary represen¬ 
tation of x and its head is on the rightmost symbol of x. The second tape 
contains the binary representation of y and its head is on the rightmost sym¬ 
bol of y. The third tape is empty and its head is at an arbitrary location. 
The Turing machine is in the start state go- 

End of the computation: The first and second tapes are empty. The third 
tape contains the binary representation of the product xy and its head is on 
the rightmost bit of xy. The Turing machine is in the final state gi. 

Hint: Use the Turing machines of Exercises 4.5 and 4.7. 


4.9 Construct a Turing machine with one tape, that receives as input a string 
of the form l n for some integer n > 0: thus, the input is a string of n many 
Is. The output of the Turing machine is the string l n Dl n . Thus, this Turing 
machine makes a copy of its input. 

The input alphabet is E = {1} and the tape alphabet is Y = {!,□}. 

Start of the computation: The tape contains a string of the form l n , for 
some integer n > 0, the tape head is on the leftmost symbol, and the Turing 
machine is in the start state. For example, if n — 4, the tape looks as follows: 

l D l D l D li|i|i|if 5 T 5 T 5 1 ••• 


End of the computation: The tape contains the string l n Dl”, the tape 
head is on the □ in the middle of this string, and the Turing machine is in 
the final state. For our example, the tape looks as follows: 

■ l D l D l D li|i|i|il D li|i|i|il D l D lU 


The Turing machine in this exercise does not have an accept state or a 
reject state; instead, it has a final state. As soon as this state is entered, the 
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Turing machine terminates. At termination, the contents of the tape is the 
output of the Turing machine. 
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Chapter 5 

Decidable and Undecidable 
Languages 


We have seen in Chapter 4 that Turing machines form a model for “everything 
that is intuitively computable”. In this chapter, we consider the limitations 
of Turing machines. That is, we ask ourselves the question whether or not 
“everything” is computable. As we will see, the answer is “no”. In fact, we 
will even see that “most” problems are not solvable by Turing machines and, 
therefore, not solvable by computers. 


5.1 Decidability 

In Chapter 4, we have defined when a Turing machine accepts an input string 
and when it rejects an input string. Based on this, we define the following 
class of languages. 

Definition 5.1.1 Let E be an alphabet and let ACE* be a language. We 
say that A is decidable , if there exists a Turing machine M, such that for 
every string w G E*, the following holds: 

1. If w G A, then the computation of the Turing machine M, on the input 
string w, terminates in the accept state. 

2. If w ^ A, then the computation of the Turing machine M, on the input 
string w, terminates in the reject state. 
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In other words, the language A is decidable, if there exists an algorithm 
that (i) terminates on every input string w, and (ii) correctly tells us whether 
w e A or w ^ A. 

A language A that is not decidable is called undecidable. For such a 
language, there does not exist an algorithm that satisfies (i) and (ii) above. 

In the following subsections, we will give some examples of decidable and 
undecidable languages. These examples involve languages A whose elements 
are pairs of the form ( C,w ), where C is some computation model (for ex¬ 
ample, a deterministic finite automaton) and w is a string over the alphabet 
S. The pair ( C , w) is in the language A if and only if the string w is in the 
language of the computation model C. For different computation models C , 
we will ask the question whether A is decidable, i.e., whether an algorithm 
exists that decides, for any input (C,w), whether or not this input belongs 
to the language A. Since the input to any algorithm is a string over some 
alphabet, we must encode the pair (C, re) as a string. In all cases that we 
consider, such a pair can be described using a finite amout of text. Therefore, 
we assume, without loss of generality, that binary strings are used for these 
encodings. Throughout the rest of this section, we will denote the binary 
encoding of a pair (C, w) by 

(O). 

5.1.1 The language Aqfa 

We define the following language: 

Abfa — {(M,w): M is a deterministic finite automaton that 
accepts the string w}. 

Keep in mind that (M, w) denotes the binary string that forms an en¬ 
coding of the finite automaton M and the string w that is given as input to 

M. 

We claim that the language A dfa is decidable. In order to prove this, 
we have to construct an algorithm with the following property, for any given 
input string u: 

• If u is the encoding of a deterministic finite automaton M and a string 
w (i.e., u is in the correct format ( M,w )), and if M accepts w, then 
the algorithm terminates in its accept state. 

• In all other cases, the algorithm terminates in its reject state. 
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An algorithm that exactly does this, is easy to obtain: On input u, the algo¬ 
rithm first checks whether or not u encodes a deterministic finite automaton 
M and a string w. If this is not the case, then it terminates and rejects 
the input string u. Otherwise, the algorithm “constructs” M and w, and 
then simulates the computation of M on the input string w. If M accepts 
w, then the algorithm terminates and accepts the input string u. If M does 
not accept w, then the algorithm terminates and rejects the input string u. 
Thus, we have proved the following result: 

Theorem 5.1.2 The language A dfa is decidable. 

5.1.2 The language Anfa 

We define the following language: 

A nfa — {(Tf, w ): M is a nondeterministic finite automaton that 
accepts the string w}. 

To prove that this language is decidable, consider the algorithm that 
does the following: On input u, the algorithm first checks whether or not 
u encodes a nondeterministic finite automaton M and a string w. If this is 
not the case, then it terminates and rejects the input string u. Otherwise, 
the algorithm constructs M and w. Since a computation of M (on input w) 
is not unique, the algorithm first converts M to an equivalent deterministic 
finite automaton N. Then, it proceeds as in Section 5.1.1. 

Observe that the construction for converting a nondeterministic finite au¬ 
tomaton to a deterministic finite automaton (see Section 2.5) is algorithmic, 
in the sense that it can be described by an algorithm. Because of this, the 
algorithm described above is a valid algorithm; it accepts all strings u that 
are in A nfa , and it rejects all strings u that are not in A nfa . Thus, we have 
proved the following result: 

Theorem 5.1.3 The language A nfa is decidable. 

5.1.3 The language Aqfg 

We define the following language: 

A cfg = {(G,w) : G is a context-free grammar such that w e L(G)}. 
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We claim that this language is decidable. In order to prove this claim, con¬ 
sider a string u that encodes a context-free grammar G = ( V , E, S, R ) and a 
string w e E*. Deciding whether or not w e L(G) is equivalent to deciding 
whether or not S A w. A first idea to decide this is by trying all possible 
derivations that start with the start variable S and that use rules of R. The 
problem is that, in case w <fL L(G), it is not clear how many such derivations 
have to be checked before we can be sure that w is not in the language of 
G: If w £ L(G), then it may be that w can be derived from S, only by first 
deriving a very long string, say v, and then use rules to shorten it so as to 
obtain the string w. Since there is no obvious upper bound on the length of 
the string v, we have to be careful. 

The trick is to do the following. First, convert the grammar G to an 
equivalent grammar G' in Chomsky normal form. (The construction given 
in Section 3.4 can be described by an algorithm.) Let n be the length of the 
string w. Then, if w E L(G) = L(G"), any derivation of w in G", from the 
start variable of G', consists of exactly 2n — 1 steps (where a “step” is defined 
as applying one rule of G'). Hence, we can decide whether or not w e L(G), 
by trying all possible derivations, in G', consisting of 2n — 1 steps. If one of 
these (finite number of) derivations leads to the string w, then w e L(G). 
Otherwise, w 0 L(G). Thus, we have proved the following result: 

Theorem 5.1.4 The language A C fg %s decidable. 

In fact, the arguments above imply the following result: 

Theorem 5.1.5 Every context-free language is decidable. 

Proof. Let E be an alphabet and let A C E* be an arbitrary context-free 
language. There exists a context-free grammar in Chomsky normal form, 
whose language is equal to A. Given an arbitrary string w 6 E*, we have 
seen above how we can decide whether or not w can be derived from the 
start variable of this grammar. I 


5.1.4 The language Atm 

After having seen the languages A dfa , A nfa , and A CF g , it is natural to 
consider the language 

Atm = { {M, w) : M is a Turing machine that accepts the string w). 
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We will prove that this language is undecidable. Before we give the proof, 
let us mention what this means: 

There is no algorithm that, when given an arbitrary algorithm M 
and an arbitrary input string w for M, decides in a finite amount 
of time, whether or not M accepts w. 

The proof of the claim that A T m is undecidable is by contradiction. Thus, 
we assume that A TM is decidable. Then there exists a Turing machine H 
that has the following property. For every input string (M, w) for H : 

• If ( M,w ) 6 A T m (he., M accepts w), then H terminates in its accept 
state. 

• If ( M,w) A T m (he., M rejects w or M does not terminate on input 
w ), then H terminates in its reject state. 

• In particular, H terminates on any input (M,w). 

We construct a new Turing machine D, that does the following: On input 
(M), the Turing machine D uses H as a subroutine to determine what M 
does when it is given its own description as input. Once D has determined 
this information, it does the opposite from what H does. 

Turing machine D: On input (M), where M is a Turing machine, 

the new Turing machine D does the following: 

Step 1: Run the Turing machine H on the input (M, (M)). 

Step 2: 

• If H terminates in its accept state, then D terminates in its 
reject state. 

• If H terminates in its reject state, then D terminates in its 
accept state. 


First observe that this new Turing machine D terminates on any input 
string (M), because H terminates on every input. Next observe that, for any 
input string ( M) for D: 

• If (M, ( M )} e A T m (he., M accepts (M)), then D terminates in its 
reject state. 
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• If (M, (M)) tfL A tm (i.e., M rejects (M) or M does not terminate on 
input (M)), then D terminates in its accept state. 

This means that for any string ( M ): 

• If M accepts (M), then D rejects (M). 

• If M rejects ( M) or M does not terminate on input (M), then D 
accepts ( M). 

We now consider what happens if we give the Turing machine D the string 
(D) as input, i.e., we take M = D: 

• If D accepts (D), then D rejects (D). 

• If D rejects ( D) or D does not terminate on input (D) , then D accepts 
(D). 

Since D terminates on every input string, this means that 

• If D accepts (.D), then D rejects (D). 

• If D rejects {D), then D accepts (D). 

This is clearly a contradiction. Therefore, the Turing machine H that decides 
the language Atm cannot exist and, thus, A TM is undecidable. We have 
proved the following result: 

Theorem 5.1.6 The language A T m is undecidable. 

5.1.5 The Halting Problem 

We define the following language: 

Halt = {(P,w): P is a Java program that terminates on 
the input string w}. 

Theorem 5.1.7 The language Halt is undecidable. 

Proof. The proof is by contradiction. Thus, we assume that the language 
Halt is decidable. Then there exists a Java program H that takes as input a 
string of the form (P, w ), where P is an arbitrary Java program and w is an 
arbitrary input for P. The program H has the following property: 
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• If ( P,w ) G Halt (i.e., program P terminates on input w), then H 
outputs true. 

• If (P, w ) 0 Halt (i.e., program P does not terminate on input w), then 
H outputs false. 

• In particular, H terminates on any input (P, w). 

We will write the output of H as P(P, w ). Moreover, we will denote by P(w) 
the computation obtained by running the program P on the input w. Hence, 

„, p s _ J true if P(w) terminates, 

' ’ ' ( false if P(w) does not terminate. 

Consider the following algorithm Q, which takes as input the encoding 
(P) of an arbitrary Java program P: 

Algorithm Q((P)): 

while H(P, ( P )) = true 
do have a beer 

endwhile 


Since H is a Java program, this new algorithm Q can also be written as 
a Java program. Observe that 

Q{(P}) terminates if and only if H(P, (P )) = false. 

This means that for every Java program P, 

Q((P)) terminates if and only if P({P}) does not terminate. (5.1) 

What happens if we run the Java program Q on the input string (Q)7 
In other words, what happens if we run Q((Q))? Then, in (5.1), we have to 
replace all occurrences of P by Q. Hence, 

Q((Q)) terminates if and only if Q((Q)) does not terminate. 

This is obviously a contradiction, and we can conclude that the Java program 
H does not exist. Therefore, the language Halt is undecidable. ■ 
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Remark 5.1.8 In this proof, we run the Java program Q on the input (Q). 
This means that the input to Q is a description of itself. In other words, we 
give Q itself as input. This is an example of what is called self-reference. If 
this does not confuse you yet, consider the following statement S: 

This statement is false. 

Is the statement S true or false ? Another example of self-reference can be 
found in Remark 5.1.8 of the textbook Introduction to Theory of Computation 
by A. Maheshwari and M. Smid. 


5.2 Countable sets 

The proofs that we gave in Sections 5.1.4 and 5.1.5 seem to be bizarre. In 
this section, we will convince you that these proofs in fact use a technique 
that you have seen in the course COMP 1805: Cantor’s Diagonalization. 

Let A and B be two sets and let / : A —> B be a function. Recall that / 
is called a bijection, if 

• / is one-to-one (or injective ), i.e., for any two distinct elements a and 
a' in A, we have f(a) ^ f(a'), and 

• / is onto (or surjective ), i.e., for each element b e B, there exists an 
element a e A, such that /(a) = b. 

The set of natural numbers is denoted by N. That is, N = {1, 2, 3,...}. 

Definition 5.2.1 Let A and B be two sets. We say that A and B have the 
same size, if there exists a bijection / : A —> B. 

Definition 5.2.2 Let A be a set. We say that A is countable, if A is finite, 
or A and N have the same size. 

In other words, if A is an infinite and countable set, then there exists a 
bijection / : N —>• A, and we can write A as 

A = {/(l),/(2),/(3),/(4),...}. 

Since / is a bijection, every element of A occurs exactly once in the set on 
the right-hand side. This means that we can number the elements of A using 
the positive integers: Every element of A receives a unique number. 
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Theorem 5.2.3 The following sets are countable: 

1. The set Z of integers: 

Z = {...,-3, -2,-1,0,1,2, 3,...}. 

2. The Cartesian product N x N: 

NxN = {(m, n) : m 6 N, n € N}. 

5. T/ie set Q of rational numbers: 

Q = {m/n :meZ,ueZ,n^0}. 

Proof. To prove that the set Z is countable, we have to give each element of 
Z a unique number in N. We obtain this numbering, by listing the elements 
of Z in the following order: 

0,1,-1,2,-2, 3,-3,4, -4,... 

In this (infinite) list, every element of Z occurs exactly once. The number of 
an element of Z is given by its position in this list. 

Formally, define the function / : N —>• Z by 

,/ s _ J n/2 if n is even, 

j[n) | —(n — l)/2 if n is odd. 

This function / is a bijection and, therefore, the sets N and Z have the same 
size. Hence, the set Z is countable. 

For the proofs of the other two claims, we refer to the course COMP 1805. 


We now use Cantor’s Diagonalization principle to prove that the set of 
real numbers is not countable: 

Theorem 5.2.4 The set M of real numbers is not countable. 

Proof. Define 

A — {x e R : 0 < x < 1}. 
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We will prove that the set A is not countable. This will imply that the set 
M is not countable, because ICl. 

The proof that A is not countable is by contradiction. So we assume that 
A is countable. Then there exists a bijection f : N —> A. Thus, for each 
n e N, f(n) is a real number between zero and one. We can write 

-4 = {/(l),/(2),/(3),...}, (5.2) 

where every element of A occurs exactly once in the set on the right-hand 
side. 

Consider the real number /(1). We can write this number in decimal 
notation as 

/(1) = 0.dn<ii2di3 .. ■, 

where each du is a digit in the set {0,1,2,..., 9}. In general, for every n <G N, 
we can write the real number /(n) as 

f(ji) = 0.d n xd n 2d n3 ..., 

where, again, each d ni is a digit in {0,1,2,..., 9}. 

We define the real number 


x — 0.did 2 d 3 ..., 


where, for each integer n > 1 , 


d n 


f 4 if d nn 7 ^ 4, 
\ 5 if d nn = 4. 


Observe that x is a real number between zero and one, i.e., x G A. Therefore, 
by (5.2), there is an element n e N, such that f(n) — x. We compare the 
n-th digits of f(n) and x: 


• The n-th digit of /(n) is equal to d nn . 


• The n-th digit of x is equal to d n . 


Since /(n) and x are equal, their n-th digits must be equal, i.e., d nn = d n . 
But, by the definition of d n , we have d nn ^ d. n . This is a contradition and, 
therefore, the set A is not countable. ■ 
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The final result of this section is the fact that for every set A, its power 
set 

V{A) = {B : B C A} 

is “strictly larger” than A. Define the function / : A —> V{A) by 

/(«) = {a}, 

for any a in A. Since / is one-to-one, we can say that V(A) is “at least as 
large as” A. 

Theorem 5.2.5 Let A be an arbitrary set. Then A and V(A) do not have 
the same size. 

Proof. The proof is by contradiction. Thus, we assume that there exists a 
bijection g : A —>• V{A). Define the set B as 

B = {a £ A : a 17 (a)}. 

Since B £ V(A) and g is a bijection, there exists an element a in A such that 
g(a) = B. 

First assume that a £ B. Since g(a) = B, we have a £ g(a). But then, 
from the definition of the set B, we have a B, which is a contradiction. 

Next assume that a B. Since g(a) = B , we have a g(a). But 
then, from the definition of the set B, we have a £ B, which is again a 
contradiction. 

We conclude that the bijection g does not exist. Therefore, A and V{A) 
do not have the same size. ■ 


5.2.1 The Halting Problem revisited 

Now that we know about countability, we give a different way to look at the 
proof in Section 5.1.5 of the fact that the language 

Halt = {(P, w): P is a Java program that terminates on 
the input string w} 

is undecidable. You should convince yourself that the proof given below 
follows the same reasoning as the one used in the proof of Theorem 5.2.4 
We first argue that the set of all Java programs is countable. Indeed, 
every Java program P can be described by a finite amount of text. In fact, 
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we have been using ( P ) to denote such a description by a binary string. For 
any integer n > 0, there are at most 2" (i.e., finitely many) Java programs 
P whose description (P) has length n. Therefore, to obtain a list of all Java 
programs, we do the following: 

• List all Java programs P whose description (P) has length 0. (Well, 
the empty string does not describe any Java program, so in this step, 
nothing happens.) 

• List all Java programs P whose description (P) has length 1. 

• List all Java programs P whose description (P) has length 2. 

• List all Java programs P whose description (P) has length 3. 

• Etcetera, etcetera. 

In this infinite list, every Java program occurs exactly once. Therefore, the 
set of all Java programs is countable. 

Consider an infinite fist 


Pi,P 2 ,P 3 ,... 


in which every Java program occurs exactly once. We modify each Java 
program P n in this list in the following way: If, for any given input, the 
computation of P n terminates, then P n returns the bit 1 (and nothing else) 
as output. 

Define the following infinite matrix T with entries t mn for nn > 1 and 
n > 1: 



1 if P m terminates on input (P n ), 

0 if P m does not terminate on input (P n ). 


Assume that the language Halt is decidable. Then there exists a Java 
program H that decides this language. We may assume that, on input (P, w ), 
H returns 1 if P terminates on input w, and 0 if P does not terminate on 
input w. Thus, for any input (P m , (P n )), the Java program H returns the 
bit t mn . 

We construct a new Java program D that does the following: 
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Algorithm D : On input (P n ), where n is a positive integer, the 
new Java program D does the following: 

Step 1: Run the Java program H on the input ( P n , (P n )), and let t 
be the output of H. (Thus, t = t nn .) 

Step 2: 

• If t = 1, then D returns 0 and terminates its computation. 

• If t = 0, then D returns 1 and terminates its computation. 


Observe that D can be written as a Java program. Therefore, there exists 
an integer n > 1 such that D = P n . Since H terminates on every input, it 
follows from the pseudocode that D terminates on every input. In particular, 
D terminates on input (P n ). Since D = P n , it follows that P n terminates on 
input (P n ). In other words, 

on input (P n ), the output of P n is 1. 

Thus, H, when given ( P n , (P n )) as input, returns 1. It then follows from the 
pseudocode that, on input ( P n ), the output of D is 0. Again using the fact 
that D = P n , this implies that 

on input (P n ), the output of P n is 0. 

Thus, we have obtained a contradiction. 

Remark 5.2.6 We defined the Java program D in such a way that, for each 
n > 1, the computation of D on input (P n ) differs from the computation of 
P n on input (P n ). Hence, for each n > 1, D P n . However, since D is a 
Java program, there must be an integer n > 1 such that D = P n . 


5.3 Rice’s Theorem 

We have seen two examples of undecidable languages: A TM and Halt. In this 
section, we prove that many languages involving Turing machines (or Java 
programs) are undecidable. 

Define 7~ to be the set of binary encodings of all Turing machines, i.e., 

T = {( M ) : M is a Turing machine with input alphabet {0,1}}. 
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Theorem 5.3.1 (Rice) Let V be a subset ofT such that 

1. V ^ 0, i.e., there exists a Turing machine M such that (M) G V, 

2. V is a proper subset of T, i.e., there exists a Turing machine N such 
that (N) tfL V, and 

3. for any two Turing machines Mi and M 2 with L(Mf) = L(Mf), 

(a) either both (Mf) and (M 2 ) are in V or 

(b) none of (Mi) and (M 2 ) is in V. 

Then the language V is undecidable. 

You can think of V as the set of all Turing machines that satisfy a certain 
property. The first two conditions state that at least one Turing machine 
satisfies this property and not all Turing machines satisfy this property. The 
third condition states that, for any Turing machine M, whether or not M 
satisfies this property only depends on the language L(M) of M. 

Here are some examples of languages that satisfy the conditions in Rice’s 
Theorem: 


V\ = {(M) : M is a Turing machine and e G L(M)}, 

V 2 = {(M) : M is a Turing machine and L(M) = {1011, 001100}}, 

V 3 = {( M) : M is a Turing machine and L(M) is a regular language}. 

You are encouraged to verify that Rice’s Theorem indeed implies that each 
of V\, R 2 , and V 3 is undecidable. 

5.3.1 Proof of Rice’s Theorem 

The strategy of the proof is as follows: Assuming that the language V is 
decidable, we show that the language 

Halt = {(M , w): M is a Turing machine that terminates on 
the input string w} 

is decidable. This will contradict Theorem 5.1.7. 
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Let Mi be a Turing machine that, for any input string, switches in its 
first computation step from its start state to its reject state. In other words, 
Mi is a Turing machine with L(M\) — 0. We assume that 

(Mi) £ V. 

(At the end of the proof, we will consider the case when (Mi) e V ) We also 
choose a Turing machine M 2 such that 

(M 2 ) e V. 

Consider a fixed Turing machine M and a fixed binary string w. We 
construct a new Turing machine T Mw that takes as input an arbitrary binary 
string x: 

Turing machine T Mw (x ): 

run Turing machine M on input w, 

if M terminates 

then run M 2 on input X) 

if M 2 terminates in the accept state 
then terminate in the accept state 
else if M 2 terminates in the reject state 
then terminate in the reject state 
endif 
endif 

endif 


We determine the language L(T Mw ) of this new Turing machine. In other 
words, we determine which strings x are accepted by T Mw . 

• Assume that M terminates on input w, i.e., ( M,w) e Halt. Then it 
follows from the pseudocode that for any string x , 

x is accepted by T Mw if and only if x is accepted by M 2 . 

Thus, L(T Mw ) = L(M 2 ). 

• Assume that M does not terminate on input w, i.e., ( M,w) ^ Halt. 
Then it follows from the pseudocode that for any string x, T Mw does 
not terminate on input x. Thus, L(T Mw ) — 0. In particular, L(T Mw ) = 
L(Mi). 
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Recall that (Mi) V, whereas (M 2 ) G V. Then the following follows from 
the third condition in Rice’s Theorem: 

• If ( M,w ) G Halt , then ( T Mw ) G V. 

• If (M, w) ^ Halt , then (T M «,) 0 P. 

Thus, we have obtained a connection between the languages V and Halt. 
This suggests that we proceed as follows. 

Assume that the language V is decidable. Let H be a Turing machine 
that decides V. Then, for any Turing machine M, 

• if (M) G V, then H accepts the string (M), 

• if (M) V, then H rejects the string (M), and 

• H terminates on any input string. 

We construct a new Turing machine H' that takes as input an arbitrary 
string (M, w), where M is a Turing machine and w is a binary string: 

Turing machine H'((M,w )): 

construct the Turing machine T Mw described above; 

run H on input (T Mw ); 

if H terminates in the accept state 

then terminate in the accept state 

else terminate in the reject state 

endif 


It follows from the pseudocode that H' terminates on any input. We 
observe the following: 

• Assume that (M, w) G Halt. Then we have seen before that (T Mw ) G V. 
Since H decides the language V, it follows that H accepts the string 
(' Tmw)■ Therefore, from the pseudocode, H' accepts the string ( M,w ). 

• Assume that ( M,w ) 0 Halt. Then we have seen before that ( Tm w ) 0 
V. Since H decides the language V. it follows that H rejects (and 
terminates on) the string (T Mw ). Therefore, from the pseudocode, H' 
rejects (and terminates on) the string (M, w). 
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We have shown that the Turing machine H' decides the language Halt. 
This is a contradiction and, therefore, we conclude that the language V is 
undecidable. 

Until now, we assumed that (Mi) ^ V. If (Mi) e V. then we repeat the 
proof with V replaced by its complement V. This revised proof then shows 
that V is undecidable. Since for every language L, 

L is decidable if and only if L is decidable, 
we again conclude that V is undecidable. 


5.4 Enumerability 

We now come to the last class of languages in this chapter: 

Definition 5.4.1 Let E be an alphabet and let A C E* be a language. We 
say that A is enumerable, if there exists a Turing machine M, such that for 
every string w e E*, the following holds: 

1. If w G A, then the computation of the Turing machine M, on the input 
string w, terminates in the accept state. 

2. If w A, then the computation of the Turing machine M, on the input 
string w , does not terminate in the accept state. That is, either the 
computation terminates in the reject state or the computation does not 
terminate. 

In other words, the language A is enumerable, if there exists an algorithm 
having the following property. If w e A, then the algorithm terminates on 
the input string w and tells us that w G A. On the other hand, if w A, 
then either (i) the algorithm terminates on the input string w and tells us 
that w $ A or (ii) the algorithm does not terminate on the input string w, 
in which case it does not tell us that w A. 

In Section 5.5, we will show where the term “enumerable” comes from. 
The following theorem follows immediately from Definitions 5.1.1 and 5.4.1. 

Theorem 5.4.2 Every decidable language is enumerable. 

In the following subsections, we will give some examples of enumerable 
languages. 
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5.4.1 Hilbert’s problem 

We have seen Hilbert’s problem in Section 4.4: Is there an algorithm that 
decides, for any given polynomial p with integer coefficients, whether or not 
p has integral roots? If we formulate this problem in terms of languages, 
then Hilbert asked whether or not the language 

Hilbert = { (p): p is a polynomial with integer coefficients 
that has an integral root} 

is decidable. As usual, (p) denotes the binary string that forms an encoding 
of the polynomial p. 

As we mentioned in Section 4.4, it was proven by Matiyasevich in 1970 
that the language Hilbert is not decidable. We claim, that this language 
is enumerable. In order to prove this claim, we have to construct an al¬ 
gorithm HILBERT with the following property: For any input polynomial p 
with integer coefficients, 

• if p has an integral root, then algorithm Hilbert will find one in a 
finite amount of time, 

• if p does not have an integral root, then either algorithm Hilbert ter¬ 
minates and tells us that p does not have an integral root, or algorithm 
Hilbert does not terminate. 

Recall that Z denotes the set of integers. Algorithm Hilbert does the 
following, on any input polynomial p with integer coefficients. Let n de¬ 
note the number of variables in p. Algorithm Hilbert tries all elements 
(xi, x 2 , ■. ■, x n ) G Z n , in a systematic way, and for each such element, it 
computes p(xi,x 2 ,..., x n ). If this value is zero, then algorithm Hilbert 
terminates and accepts the input. 

We observe the following: 

• If p G Hilbert , then algorithm Hilbert terminates and accepts p, pro¬ 
vided we are able to visit all elements (aq, x 2 ,..., x n ) G Z n in a “sys¬ 
tematic way”. 


• If p Hilbert , then p(x i, x 2 ,..., x n ) ^ 0 for all (aq, x 2 ,..., x n ) G Z n 

and, therefore, algorithm Hilbert does not terminate. 

These are exactly the requirements for the language Hilbert to be enumerable. 
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It remains to explain how we visit all elements (xi, x 2 ,..., x n ) G Z n in a 
systematic way. For any integer d > 0, let H d denote the hypercube in Z n 
with sides of length 2d that is centered at the origin. That is, H d consists 
of the set of all points (xi, x 2 ,..., x n ) in Z™, such that — d < Xi < d for all 
1 < i < n and there exists at least one index j for which Xj = d or x 3 — —d. 
We observe that H d contains a finite number of elements. In fact, if ci > 1, 
then this number is equal to (2d + l) n — (2d — l) n . The algorithm will visit 
all elements (xi,x 2 , • • •, x n ) G Z n , in the following order: First, it visits the 
origin, which is the only element of H 0 . Then, it visits all elements of H\. 
followed by all elements of H 2 , etc., etc. 

To summarize, we obtain the following algorithm, proving that the lan¬ 
guage Hilbert is enumerable: 

Algorithm Hilbert((p}): 

n := the number of variables in p; 
d := 0; 

while d > 0 

do for each (aq, x 2 ,..., x n ) G H d 
do R := p(x i, x 2 ,..., x n ); 

if R = 0 

then terminate and accept 

endif 
endfor; 
d := d+ 1 
endwhile 


Theorem 5.4.3 The language Hilbert is enumerable. 

5.4.2 The language Atm 

We have shown in Section 5.1.4 that the language 

Atm = {(M, w) : M is a Turing machine that accepts the string w}. 

is undecidable. Interestingly, this language is enumerable. In order to prove 
this, we have to construct an algorithm P having the following property, for 
any given input string u: 
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• If 


— u encodes a Turing machine M and an input string w for M (i.e., 
u is in the correct format (M, w )) and 

- ( M,w } e A T m (he., M accepts w), 

then algorithm P terminates in its accept state. 

• In all other cases, either algorithm P terminates in its reject state, or 
algorithm P does not terminate. 

On input string u = (M, w), which is in the correct format, algorithm P does 
the following: 

1. It simulates the computation of M on input w. 

2. If M terminates in its accept state, then P terminates in its accept 
state. 

3. If M terminates in its reject state, then P terminates in its reject state. 

4. If M does not terminate, then P does not terminate. 

Hence, if u — {M, w) G A TM , then M accepts w and, therefore, P accepts 
u. On the other hand, if u — (M, w) A TM , then M does not accept w. This 
means that, on input w, M either terminates in its reject state or does not 
terminate. But this implies that, on input u, P either terminates in its reject 
state or does not terminate. This proves that algorithm P has the properties 
that are needed in order to show that the language A TM is enumerable. We 
have proved the following result: 

Theorem 5.4.4 The language Atm Is enumerable. 


5.5 Where does the term “enumerable” come 
from? 

In Definition 5.4.1, we have defined what it means for a language to be 
enumerable. In this section, we will see where this term comes from. 
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Definition 5.5.1 Let E be an alphabet and let A C E* be a language. An 
enumerator for A is a Turing machine E having the following properties: 

1. Besides the standard features as in Section 4.1 , E has a print tape and 
a print state. During its computation, E writes symbols of E on the 
print tape. Each time, E enters the print state, the current string on 
the print tape is sent to the printer and the print tape is made empty. 

2. At the start of the computation, all tapes are empty and E is in the 
start state. 

3. Every string w in A is sent to the printer at least once. 

4. Every string w that is not in A is never sent to the printer. 

Thus, an enumerator E for A really enumerates all strings in the language 
A. There is no particular order in which the strings of A are sent to the 
printer. Moreover, a string in A may be sent to the printer multiple times. 
If the language A is infinite, then the Turing machine E obviously does not 
terminate; however, every string in A (and only strings in A) will be sent to 
the printer at some time during the computation. 

To give an example, let A = {0 n : n > ()}. The following Turing machine 
is an enumerator for A. 

Turing machine StringsOfZeroS: 

n 0; 

while 1 + 1 = 2 
do for i := 1 to n 

do write 0 on the print tape 
endfor; 

enter the print state; 
n n+1 

endwhile 


In the rest of this section, we will prove the following result. 

Theorem 5.5.2 A language is enumerable if and only if it has an enumer¬ 
ator. 
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For the first part of the proof, assume that the language A has an enu¬ 
merator E. We construct the following Turing machine M, which takes an 
arbitrary string w as input: 

Turing machine M{w ): 

run E ; every time E enters the print state: 
let v be the string on the print tape; 
if w — v 

then terminate in the accept state 

endif 


The Turing machine M has the following properties: 


• If w G A, then w will be sent to the printer at some time during the 
computation of E. It follows from the pseudocode that, on input w, 
M terminates in the accept state. 


• If w 0 A, then E will never sent w to the printer. It follows from the 
pseudocode that, on input w, M does not terminate. 


Thus, M satisfies the conditions in Definition 5.4.1. We conclude that the 
language A is enumerable. 

To prove the converse, we now assume that A is enumerable. Let M be 
a Turing machine that satisfies the conditions in Definition 5.4.1. 

We fix an infinite list 


of all strings in E*. For example, if E = {0,1}, then we can take this list to 
be 


e, 0,1, 00, 01,10,11, 000, 001, 010,100, Oil, 101,110, 111,... 

We construct the following Turing machine E, which takes the empty 
string as input: 
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Turing machine E: 

n := 1; 

while 1 + 1 = 2 
do for i := 1 to n 

do run M for n steps on the input string sf, 
if M accepts s* within n steps 
then write s* on the print tape; 
enter the print state 

endif 

endfor; 

n := n + 1 

endwhile 


We claim that E is an enumerator for the language A. To prove, it is 
obvious that any string that is sent to the printer by E belongs to A. 

It remains to prove that every string in A will be sent to the printer by E. 
Let w be a string in A. Then, on input u>, the Turing machine M terminates 
in the accept state. Let m be the number of steps made by M on input w. 
Let i be the index such that w = ■s l . Define n = max(m, i). Consider the 
n-th iteration of the while-loop and the i-th iteration of the for-loop. In this 
iteration, M accepts ■s l = w in m < n steps and, therefore, w is sent to the 
printer. 


5.6 Most languages are not enumerable 

In this section, we will prove that most languages that are not enumerable. 
The proof is based on the following two facts: 


• The set consisting of all enumerable languages is countable; we will 
prove this in Section 5.6.1. 


• The set consisting of all languages is not countable; we will prove this 
in Section 5.6.2. 
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5.6.1 The set of enumerable languages is countable 

We define the set £ as 

£ = {A : A C (0,1}* is an enumerable language}. 

In words, £ is the set whose elements are the enumerable languages. Every 
element of £ is an enumerable language. Hence, every element of the set £ 
is itself a set consisting of strings. 

Lemma 5.6.1 The set £ is countable. 

Proof. Let A C {0,1}* be an enumerable language. There exists a Turing 
machine T A that satisfies the conditions in Definition 5.4.1. This Turing 
machine T A can be uniquely specified by a string in English. This string can 
be converted to a binary string s A . Hence, the binary string s A is a unique 
encoding of the Turing machine T A . 

Consider the set 

S = {s A : A C {0,1}* is an enumerable language}. 

Observe that the function /:£—>■ <S, defined by f(A) = s A for each Aef, 
is a bijection. Therefore, the sets £ and S have the same size. Hence, in 
order to prove that the set £ is countable, it is sufficient to prove that the 
set S is countable. 

Why is the set S countable? For each integer n > 0, there are exactly 2 n 
binary strings of length n. Since there are binary strings that are not encod¬ 
ings of Turing machines, the set S contains at most 2 n strings of length n. 
In particular, the number of strings in S having length n is finite. Therefore, 
we obtain an infinite list of the elements of S in the following way: 

• List all strings in S having length 0. (Well, the empty string is not in 
S, so in this step, nothing happens.) 

• List all strings in S having length 1. 

• List all strings in S having length 2. 

• List all strings in S having length 3. 

• Etcetera, etcetera. 

In this infinite list, every element of S occurs exactly once. Therefore, S is 
countable. ■ 
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5.6.2 The set of all languages is not countable 

We define the set £ as 

£ = {A : A C {0,1}* is a language}. 

In words, £ is the set consisting of all languages. Every element of the set £ 
is a set consisting of strings. 

Lemma 5.6.2 The set £ is not countable. 

Proof. We define the set B as 

B = {w : w is an infinite binary sequence}. 

We claim that this set is not countable. The proof of this claim is almost 
identical to the proof of Theorem 5.2.4. We assume that the set B is count¬ 
able. Then there exists a bijection / : N —> B. Thus, for each n <E N, f(n) is 
an infinite binary sequence. We can write 

B = {/(!),/(2),/(3),...}, (5.3) 

where every element of B occurs exactly once in the set on the right-hand 
side. 

We define the infinite binary sequence w = WiW 2 w 3 ..., where, for each 
integer n > 1, 


_ ( 1 if the n-th bit of f(n) is 0, 

Wn \ 0 if the n-th bit of f (n) is 1. 

Since w 6 B, it follows from (5.3) that there is an element n e N, such that 
f(n) = w. Hence, the n-th bits of /(n) and w are equal. But, by definition, 
these n-th bits are not equal. This is a contradition and, therefore, the set 
B is not countable. 

In the rest of the proof, we will show that the sets £ and B have the same 
size. Since B is not countable, this will imply that £ is not countable. 

In order to prove that £ and B have the same size, we have to show that 
there exists a bijection 

g : £ —>• B. 
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We first observe that the set {0,1}* is countable, because for each integer 
n > 0, there are only finitely many (to be precise, exactly 2 n ) binary strings 
of length n. In fact, we can write 

{0,1}* = {e, 0,1, 00, 01,10,11, 000, 001, 010,100, Oil, 101,110, 111,.. 

For each integer n > 1, we denote by s n the rt-th string in this list. Hence, 
{0,1}* = {si,s 2 ,s 3 ,...}. (5.4) 

Now we are ready to define the bijection g : C —>• £>: Let A e C, i.e., 
A C {0,1}* is a language. We define the infinite binary sequence g(A) as 
follows: For each integer n > 1, the n-th bit of g(A) is equal to 

f 1 ifs n eA, 

{0 if s n A. 

In words, the infinite binary sequence g(A) contains a 1 exactly in those 
positions n for which the string s n in (5.4) is in the language A. 

To give an example, assume that A is the language consisting of all binary 
strings that start with 0. The following table gives the corresponding infinite 
binary sequence g(A) (this sequence is obtained by reading the rightmost 
column from top to bottom): 


{0,1}* 

A 

9(A) 

e 

not in A 

0 

0 

in A 

1 

1 

not in A 

0 

00 

in A 

1 

01 

in A 

1 

10 

not in A 

0 

11 

not in A 

0 

000 

in A 

1 

001 

in A 

1 

010 

in A 

1 

100 

not in A 

0 

Oil 

in A 

1 

101 

not in A 

0 

110 

not in A 

0 

111 

not in A 

0 
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The function g defined above has the following properties: 

• If A and A! are two different languages in £, then g(A) ^ g(A'). 

• For every infinite binary sequence w in B, there exists a language A in 
£, such that g(A) — w. 

This means that the function g is a bijection from £ to B. ■ 


5.6.3 There are languages that are not enumerable 

We have proved that the set 

S = {A : A C {0,1}* is an enumerable language} 
is countable, whereas the set 

£ = {A : A C {0,1}* is a language} 

is not countable. This means that there are “more” languages in £ than 
there are in £, proving the following result: 

Theorem 5.6.3 There exist languages that are not enumerable. 

The proof given above shows the existence of languages that are not 
enumerable. However, the proof does not give us a specific example of a 
language that is not enumerable. In the next sections, we will see examples 
of such languages. 

5.7 The relationship between decidable and 
enumerable languages 

We know from Theorem 5.4.2 that every decidable language is enumerable. 
On the other hand, we know from Theorems 5.1.6 and 5.4.4 that the converse 
is not true. The following result should not come as a surprise: 

Theorem 5.7.1 Let £ be an alphabet and let A C £* be a language. Then, 
A is decidable if and only if both A and its complement A are enumerable. 
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Proof. We first assume that A is decidable. Then, by Theorem 5.4.2, A 
is enumerable. Since A is decidable, it is not difficult to see that A is also 
decidable. Then, again by Theorem 5.4.2, A is enumerable. 

To prove the converse, we assume that both A and A are enumerable. 
Since A is enumerable, there exists a Turing machine Ml, such that for any 
string w£E*, the following holds: 

• If w e A, then the computation of Mi, on the input string w, terminates 
in the accept state of Mi. 

• If w A, then the computation of Mi, on the input string w, terminates 
in the reject state of Mi or does not terminate. 

Similarly, since A is enumerable, there exists a Turing machine M 2 , such that 
for any string w £ E*, the following holds: 

• If w & A, then the computation of M 2 , on the input string w, terminates 
in the accept state of M 2 . 

• If w 0 A, then the computation of M 2 , on the input string w, terminates 
in the reject state of M 2 or does not terminate. 

We construct a two-tape Turing machine M: 

Two-tape Turing machine M: For any input string w £ £*, M 
does the following: 

• M simulates the computation of Mi, on input w, on the first 
tape, and, simultaneously, it simulates the computation of M 2 , 
on input w, on the second tape. 

• If the simulation of Mi terminates in the accept state of Mi, 
then M terminates in its accept state. 

• If the simulation of M 2 terminates in the accept state of M 2 , 
then M terminates in its reject state. 


Observe the following: 

• If w e A, then Mi terminates in its accept state and, therefore, M 
terminates in its accept state. 
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• If w A, then M 2 terminates in its accept state and, therefore, M 
terminates in its reject state. 

We conclude that the Turing machine M accepts all strings in A , and rejects 
all strings that are not in A. This proves that the language A is decidable. 


We now use Theorem 5.7.1 to give examples of languages that are not 
enumerable: 

Theorem 5.7.2 The language A T m is not enumerable. 

Proof. We know from Theorems 5.4.4 and 5.1.6 that the language Atm is 
enumerable but not decidable. Combining these facts with Theorem 5.7.1 
implies that the language A TM is not enumerable. ■ 

The following result can be proved in exactly the same way: 

Theorem 5.7.3 The language Halt is not enumerable. 

5.8 A language A such that both A and A are 
not enumerable 

In Theorem 5.7.2, we have seen that the complement of the language A TM 
is not enumerable. In Theorem 5.4.4, however, we have shown that the 
language A T m itself is enumerable. In this section, we consider the language 

EQtm = {(Mi, M 2 ): Mi and M 2 are Turing machines 
and L(Mi) = L(M 2 )}. 

We will show the following result: 

Theorem 5.8.1 Both EQ TM and its complement EQ TM are not enumer¬ 
able. 


5.8.1 EQ tm is not enumerable 

Consider a fixed Turing machine M and a fixed binary string w. We construct 
a new Turing machine Tmw that takes as input an arbitrary binary string x: 
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Turing machine T Mw {x): 

run Turing machine M on input w, 
terminate in the accept state 


We determine the language L(Tm w ) of this new Turing machine. In other 
words, we determine which strings x are accepted by T Mw . 

• Assume that M terminates on input w, i.e., (M, w) 6 Halt. Then it 
follows from the pseudocode that every string x is accepted by T Mw - 
Thus, L(T Mw ) = {0,1}*. 

• Assume that M does not terminate on input w, i.e., ( M,w ) e Halt. 
Then it follows from the pseudocode that, for any string x, T Mw does 
not terminate on input x. Thus, L(T Mw ) = 0. 

Assume that the language EQ TM is enumerable. We will show that Halt 
is enumerable as well, which will contradict Theorem 5.7.3. 

Since EQ TM is enumerable, there exists a Turing machine H having the 
following property, for any two Turing machines M\ and M 2 : 

• If L(Mi) = L(M 2 ), then, on input (Mi, M 2 ), H terminates in the accept 
state. 

• If L(M\) 7^ L(M 2 ), then, on input (Mi,M 2 ), H either terminates in 
the reject state or does not terminate. 

We construct a new Turing machine H' that takes as input an arbitrary 
string (M, w }, where M is a Turing machine and w is a binary string: 

Turing machine H'((M,w)): 

construct a Turing machine Mi that rejects every input string; 
construct the Turing machine T Mw described above; 
run H on input (M], T Mv ) ; 
if H terminates in the accept state 
then terminate in the accept state 
else if H terminates in the reject state 
then terminate in the reject state 
endif 
endif 
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We observe the following: 

• Assume that (M, w) 6 Halt. Then we have seen before that L(Tm w ) — 
0. By our choice of Mi, we have L(Mi) = 0 as well. Therefore, H 
accepts (and terminates on) the input (Mi,T M w ). It follows from the 
pseudocode that H' accepts (and terminates on) the string (M, w). 

• Assume that ( M,w ) 0 Halt, i.e., ( M,w) e Halt. Then we have seen 
before that L(T Mw ) 0 = L(Mi). Therefore, on input (Mi, T Mw ), H 
either terminates in the reject state or does not terminate. It follows 
from the pseudocode that, on input (M, w), H' either terminates in the 
reject state or does not terminate. 

Thus, the Turing machine H' has the properties needed to show that 
the language Halt is enumerable. This is a contradiction and, therefore, we 
conclude that the language EQ TM is not enumerable. 


5.8.2 EQ tm is not enumerable 

This proof is symmetric to the one in Section 5.8.1. For a fixed Turing 
machine M and a fixed binary string w, we will use the same Turing machine 
Tmw as in Section 5.8.1. 

Assume that the language EQ TM is enumerable. We will show that Halt 
is enumerable as well, which will contradict Theorem 5.7.3. 

Since EQ TM is enumerable, there exists a Turing machine H having the 
following property, for any two Turing machines M\ and M 2 : 

• If L(Mi) ^ L(M 2 ), then, on input (Mi, M 2 ), H terminates in the accept 
state. 

• If L(Mi ) = L(M 2 ), then, on input ( Mi,M 2 ), H either terminates in 
the reject state or does not terminate. 

We construct a new Turing machine H' that takes as input an arbitrary 
string (M, w ), where M is a Turing machine and w is a binary string: 
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Turing machine H'((M,w )): 

construct a Turing machine Mi that accepts every input string; 
construct the Turing machine T Mw of Section 5.8.1; 
run H on input (M i, T Mw ); 
if H terminates in the accept state 
then terminate in the accept state 
else if H terminates in the reject state 
then terminate in the reject state 
endif 
endif 


We observe the following: 

• Assume that (M, w) e Halt. Then we have seen before that L(T Mw ) — 
0. Thus, by our choice of Mi, we have L(T Mw ) ^ L(M X ). Therefore, H 
accepts (and terminates on) the input ( Mi,T Mw ). It follows from the 
pseudocode that H' accepts (and terminates on) the string (M, w). 

• Assume that (M,w) 0 Halt. Then L(T Mw ) — {0,1}* = L(Mi) and, on 
input (Mi,T M w ), H either terminates in the reject state or does not 
terminate. It follows from the pseudocode that, on input ( M,w ), H' 
either terminates in the reject state or does not terminate. 

Thus, the Turing machine H' has the properties needed to show that 
the language Halt is enumerable. This is a contradiction and, therefore, we 
conclude that the language EQ TM is not enumerable. 


Exercises 

5.1 Prove that the language 

{w £ {0,1}* : w is the binary representation of 2 n for some n > 0} 

is decidable. In other words, construct a Turing machine that gets as input 
an arbitrary number x e N, represented in binary as a string w, and that 
decides whether or not a; is a power of two. 

5.2 Let F be the set of all functions / : N —>• N. Prove that F is not 
countable. 
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5.3 A function / : N — > N is called computable , if there exists a Turing 
machine, that gets as input an arbitrary positive integer n, written in binary, 
and gives as output the value of f(n), again written in binary. This Turing 
machine has a final state. As soon as the Turing machine enters this final 
state, the computation terminates, and the output is the binary string that 
is written on its tape. 

Prove that there exist functions / : N —>• N that are not computable. 

5.4 Let n be a fixed positive integer, and let k be the number of bits in the 
binary representation of n. (Hence, k = 1 + [lognj.) Construct a Turing 
machine with one tape, tape alphabet {0,1, □}, and exactly k + 1 states 
qo,qi, ■ ■ ■ ,q k , that does the following: 

Start of the computation: The tape is empty, i.e., every cell of the tape 
contains □, and the Turing machine is in the start state q 0 . 

End of the computation: The tape contains the binary representation of 
the integer n, the tape head is on the rightmost bit of the binary represen¬ 
tation of n, and the Turing machine is in the final state q k . 

The Turing machine in this exercise does not have an accept state or a 
reject state; instead, it has a final state q k . As soon as state q k is entered, 
the Turing machine terminates. 

5.5 Give an informal description (in plain English) of a Turing machine 
with three tapes, that gets as input the binary representation of an arbitrary 
integer m > 1, and returns as output the unary representation of m. 

Start of the computation: The first tape contains the binary representa¬ 
tion of the input m. The other two tapes are empty (i.e., contain only Ds). 
The Turing machine is in the start state. 

End of the computation: The third tape contains the unary representation 
of m, i.e., a string consisting of m many ones. The Turing machine is in the 
final state. 

The Turing machine in this exercise does not have an accept state or a 
reject state; instead, it has a final state. As soon as this final state is entered, 
the Turing machine terminates. 

Hint: Use the second tape to maintain a string of ones, whose length is 
a power of two. 
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5.6 In this exercise, you are asked to prove that the busy beaver function 
BB : N —>• N is not computable. 

For any integer n > 1, we define TM n to be the set of all Turing machines 
M, such that 

• M has one tape, 

• M has exactly n states, 

• the tape alphabet of M is {0,1, □}, and 

• M terminates, when given the empty string e as input. 

For every Turing machine M 6 TM n , we define f(M) to be the number of 
ones on the tape, after the computation of M, on the empty input string, 
has terminated. 

The busy beaver function BB : N —>• N is defined as 

BB(n) := max{/(M) : M € TM n }, for every n > 1. 

In words, BB(n) is the maximum number of ones that any Turing machine 
with n states can produce, when given the empty string as input, and as¬ 
suming the Turing machine terminates on this input. 

Prove that the function BB is not computable. 

Hint: Assume that BB is computable. Then there exists a Turing ma¬ 
chine M that, for any given n > 1, computes the value of BB(n). Fix a large 
integer n > 1. Define (in plain English) a Turing machine that, when given 
the empty string as input, terminates and outputs a string consisting of more 
than BB(n) many ones. Use Exercises 5.4 and 5.5 to argue that there exists 
such a Turing machine having O(logn) states. Then, if you assume that n 
is large enough, the number of states is at most n. 

5.7 Since the set 


T = {M : M is a Turing machine} 
is countable, there is an infinite list 

M 1 ,M 2 ,M 3 ,M 4 ,..., 

such that every Turing machine occurs exactly once in this list. 
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For any positive integer n, let (n) denote the binary representation of n; 
observe that (n) is a binary string. 

Let A be the language defined as 

A = {( n ) : the Turing machine M n terminates on the input string (n), 
and it rejects this string}. 

Prove that the language A is undecidable. 

5.8 Consider the three languages 

Empty = {(M) : M is a Turing machine for which L(M) = 0}, 

UselessState = {( M , q): M is a Turing machine, q is a state of M, 

for every input string w , the computation of M on 
input w never visits state q}, 

and 

EQtm = { (Mi , M 2 ): Mi and M 2 are Turing machines 
and L(M 1 ) = L(M 2 )}. 

• Use Rice’s Theorem to show that Empty is undecidable. 

• Use the first part to show that UselessState is undecidable. 

• Use the first part to show that EQ TM is undecidable. 

5.9 Consider the language 

REGtm = {(M) : M is a Turing machine whose language L{M) is regular}. 

Use Rice’s Theorem to prove that REG T m is undecidable. 

5.10 We have seen in Section 5.1.4 that the language 

Atm = {(M, w) : M is a Turing machine that accepts w} 

is undecidable. Consider the language REGtm of Exercise 5.9. The questions 
below will lead you through a proof of the claim that the language REG T m 
is undecidable. 
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Consider a fixed Turing machine M and a fixed binary string w. We 
construct a new Turing machine T Mw that takes as input an arbitrary binary 
string x: 

Turing machine T Mw {x ): 

if x = 0 n l” for some n > 0 
then terminate in the accept state 
else run M on the input string w; 

if M terminates in the accept state 
then terminate in the accept state 
else if M terminates in the reject state 
then terminate in the reject state 
endif 
endif 
endif 


Answer the following two questions: 

• Assume that M accepts the string w. What is the language L(T Mw ) of 
the new Turing machine Tmw ? 

• Assume that M does not accept the string w. What is the language 
L(T Mw ) of the new Turing machine T Mw ? 


The goal is to prove that the language REGtm is undecidable. We will 
prove this by contradiction. Thus, we assume that R is a Turing machine 
that decides REG T m ■ Recall what this means: 

• If M is a Turing machine whose language is regular, then R, when 
given (M) as input, will terminate in the accept state. 

• If M is a Turing machine whose language is not regular, then R , when 
given (M) as input, will terminate in the reject state. 

We construct a new Turing machine R' which takes as input an arbitrary 
Turing machine M and an arbitrary binary string w: 
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Turing machine R'((M,w )): 

construct the Turing machine T Mw described above; 
run R on the input ( T Mw ); 
if R terminates in the accept state 
then terminate in the accept state 
else if R terminates in the reject state 
then terminate in the reject state 
endif 
endif 


Prove that M accepts w if and only if R' (when given ( M,w) as input), 
terminates in the accept state. 

Now finish the proof by arguing that the language REGtm is undecidable. 

5.11 A Java program P is called a Hello-World-program, if the following is 
true: When given the empty string e as input, P outputs the string Hello 
World and then terminates. (We do not care what P does when the input 
string is non-empty.) 

Consider the language 

HW = {(P) : P is a Hello-World-program}. 

The questions below will lead you through a proof of the claim that the 
language HW is undecidable. 

Consider a fixed Java program P and a fixed binary string w. We write 
a new Java program J Pw which takes as input an arbitrary binary string x: 

Java program J Pw {x)\ 

run P on the input w, 
print Hello World 


• Argue that P terminates on input w if and only if ( J Pw ) <G HW. 

The goal is to prove that the language HW is undecidable. We will prove this 
by contradiction. Thus, we assume that H is a Java program that decides 
HW. Recall what this means: 
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• If P is a Hello-World-program, then H, when given ( P) as input, will 
terminate in the accept state. 

• If P is not a Hello-World-program, then H, when given ( P) as input, 
will terminate in the reject state. 

We write a new Java program H' which takes as input the binary encoding 
(P,w) of an arbitrary Java program P and an arbitrary binary string w: 

Java program H'((P,w )): 

construct the Java program J Pw described above; 

run H on the input ( J Pw ); 

if H terminates in the accept state 

then terminate in the accept state 

else terminate in the reject state 

endif 


Argue that the following are true: 

• For any input ( P,w ), H' terminates. 

• If P terminates on input w, then H' (when given ( P,w ) as input), 
terminates in the accept state. 

• If P does not terminate on input w, then H' (when given ( P,w ) as 
input), terminates in the reject state. 

Now finish the proof by arguing that the language HW is undecidable. 

5.12 Prove that the language Halt, see Section 5.1.5, is enumerable. 

5.13 We define the following language: 

L = {u : u = (0, M, w) for some (M, w) e A TM , 

or u = (1, M, w) for some {M, w) £ A TM } . 

Prove that neither L nor its complement L is enumerable. 

Hint: There are two ways to solve this exercise. In the first solution, (i) 
you assume that L is enumerable, and then prove that Atm is decidable, and 
(ii) you assume that L is enumerable, and then prove that A T m is decidable. 
In the second solution, (i) you assume that L is enumerable, and then prove 
that A T m is enumerable, and (ii) you assume that L is enumerable, and then 
prove that A T m is enumerable. 
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Complexity Theory 


In the previous chapters, we have considered the problem of what can be 
computed by Turing machines (i.e., computers) and what cannot be com¬ 
puted. We did not, however, take the efficiency of the computations into 
account. In this chapter, we introduce a classification of decidable languages 
A, based on the running time of the “best” algorithm that decides A. That 
is, given a decidable language A, we are interested in the “fastest” algorithm 
that, for any given string w , decides whether or not w G A. 


6.1 The running time of algorithms 


Let M be a Turing machine, and let w be an input string for M. We define 
the running time t M (w) of M on input w as 

t M {w) := the number of computation steps made by M on input w. 

As usual, we denote by \w\, the number of symbols in the string w. We 
denote the set of non-negative integers by N 0 . 


Definition 6.1.1 Let E be an alphabet, let T : N 0 —> N 0 be a function, let 
A C E* be a decidable language, and let F : E* —>■ E* be a computable 
function. 


• We say that the Turing machine M decides the language A in time T, 
if 


Lu(w) < T(\w\) 


for all strings w in E*. 
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• We say that the Turing machine M computes the function F in time 
T, if 

t M {w) < T(\w\) 

for all strings w e £*. 

In other words, the “running time function” T is a function of the length 
of the input, which we usually denote by n. For any n, the value of T (n) is 
an upper bound on the running time of the Turing machine M, on any input 
string of length n. 

To give an example, consider the Turing machine of Section 4.2.1 that 
decides, using one tape, the language consisting of all palindromes. The tape 
head of this Turing machine moves from the left to the right, then back to 
the left, then to the right again, back to the left, etc. Each time it reaches 
the leftmost or rightmost symbol, it deletes this symbol. The running time 
of this Turing machine, on any input string of length n, is 

0(1 + 2 + 3 + ... + n) = 0(n 2 ). 

On the other hand, the running time of the Turing machine of Section 4.2.2, 
which also decides the palindromes, but using two tapes instead of just one, 
is 0(n). 

In Section 4.4, we mentioned that all computation models listed there are 
equivalent, in the sense that if a language can be decided in one model, it 
can be decided in any of the other models. We just saw, however, that the 
language consisting of all palindromes allows a faster algorithm on a two- 
tape Turing machine than on one-tape Turing machines. (Even though we 
did not prove this, it is true that Q(n 2 ) is a lower bound on the running 
time to decide palindromes on a one-tape Turing machine.) The following 
theorem can be proved. 

Theorem 6.1.2 Let A be a language (resp. let F be a function) that can be 
decided (resp. computed) in time T by an algorithm of type M. Then there is 
an algorithm of type N that decides A (resp. computes F) in time T', where 


M 

N 

r 

k-tape Turing machine 

one-tape Turing machine 

0(T 2 ) 

one-tape Turing machine 

Java program 

0{T 2 ) 

Java program 

k-tape Turing machine 

0(T 4 ) 
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6.2 The complexity class P 

Definition 6.2.1 We say that algorithm M decides the language A (resp. 
computes the function F) in polynomial time, if there exists an integer k > 1, 
such that the running time of M is 0{n k ), for any input string of length n. 

It follows from Theorem 6.1.2 that this notion of “polynomial time” does 
not depend on the model of computation: 

Theorem 6.2.2 Consider the models of computation “Java program”, “k- 
tape Turing machine”, and “one-tape Turing machine”. If a language can 
be decided (resp. a function can be computed) in polynomial time in one of 
these models, then it can be decided (resp. computed) in polynomial time in 
all of these models. 

Because of this theorem, we can define the following two complexity 
classes: 


P := {^4 : the language A is decidable in polynomial time}, 

and 


FP := {F : the function F is computable in polynomial time}. 

6.2.1 Some examples 

Palindromes 

Let Pal be the language 

Pal := {w E {a, b}* : w is a palindrome}. 

We have seen that there exists a one-tape Turing machine that decides Pal 
in 0(n 2 ) time. Therefore, Pal e P. 

Some functions in FP 

The following functions are in the class FP: 

• Fi : N 0 —>• N 0 defined by Ffx) := x + 1, 

• F 2 : Nq —>• N 0 defined by F 2 (x, y) := x + y, 

• F 3 : Mg —>• N 0 defined by F :i (x, y) := xy. 
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b 




Figure 6.1: The graph G\ is 2-colorable; r stands for red; b stands for blue. 
The graph G 2 is not 2-colorable. 


Context-free languages 

We have shown in Section 5.1.3 that every context-free language is decid¬ 
able. The algorithm presented there, however, does not run in polynomial 
time. Using a technique called dynamic programming (which you will learn 
in COMP 3804), the following result can be shown: 

Theorem 6.2.3 Let E be an alphabet, and let A C E* be a context-free 
language. Then AeP. 

Observe that, obviously, every language in P is decidable. 

The 2-coloring problem 

Let G be a graph with vertex set V and edge set E. We say that G is 
2-colorable , if it is possible to give each vertex of Fa color such that 

1. for each edge (it, v) e E, the vertices u and v have different colors, and 

2. only two colors are used to color all vertices. 

See Figure 6.1 for two examples. We define the following language: 

2 Color := {(G) : the graph G is 2-colorable}, 
where (G) denotes the binary string that encodes the graph G. 
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We claim that 2 Color e P. In order to show this, we have to construct an 
algorithm that decides in polynomial time, whether or not any given graph 
is 2-color able. 

Let G be an arbitrary graph with vertex set V = {1,2,..., m}. The edge 
set of G is given by an adjacency matrix. This matrix, which we denote by 
E, is a two-dimensional array with m rows and m columns. For all i and j 
with 1 < i < m and 1 < j < m, we have 


E(iJ) = { 


1 if (i,j) is an edge of G, 
0 otherwise. 


The length of the input G, i.e., the number of bits needed to specify G, is 
equal to m 2 =: n. We will present an algorithm that decides, in O(n) time, 
whether or not the graph G is 2-colorable. 

The algorithm uses the colors red and blue. It gives the first vertex the 
color red. Then, the algorithm considers all vertices that are connected by 
an edge to the first vertex, and colors them blue. Now the algorithm is done 
with the first vertex; it marks this first vertex. 

Next, the algorithm chooses a vertex i that already has a color, but that 
has not been marked. Then it considers all vertices j that are connected by 
an edge to z. If j has the same color as z, then the input graph G is not 
2-colorable. Otherwise, if vertex j does not have a color yet, the algorithm 
gives j the color that is different from z’s color. After having done this for 
all neighbors j of z, the algorithm is done with vertex z, so it marks z. 

It may happen that there is no vertex z that already has a color but that 
has not been marked. (In other words, each vertex z that is not marked does 
not have a color yet.) In this case, the algorithm chooses an arbitrary vertex 
z having this property, and colors it red. (This vertex z is the first vertex in 
its connected component that gets a color.) 

This procedure is repeated until all vertices of G have been marked. 

We now give a formal description of this algorithm. Vertex z has been 
marked , if 

1. z has a color, 

2. all vertices that are connected by an edge to z have a color, and 

3. the algorithm has verified that each vertex that is connected by an edge 
to z has a color different from z’s color. 
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The algorithm uses two arrays /(I... m) and a(l... m), and a variable 
M. The value of f{i) is equal to the color (red or blue) of vertex v, if i does 
not have a color yet, then f(i) — 0. The value of a(i) is equal to 

,.. J 1 if vertex i has been marked, 
a { 0 otherwise. 

The value of M is equal to the number of marked vertices. The algorithm 
is presented in Figure 6.2. You are encouraged to convince yourself of the 
correctness of this algorithm. That is, you should convince yourself that this 
algorithm returns YES if the graph G is 2-colorable, whereas it returns NO 
otherwise. 

What is the running time of this algorithm? First we count the number 
of iterations of the outer while-loop. In one iteration, either M increases by 
one, or a vertex i, for which a(i) = 0, gets the color red. In the latter case, 
the variable M is increased during the next iteration of the outer while-loop. 
Since, during the entire outer while-loop, the value of M is increased from 
zero tom, it follows that there are at most 2m iterations of the outer while- 
loop. (In fact, the number of iterations is equal to m plus the number of 
connected components of G minus one.) 

One iteration of the outer while-loop takes 0{m ) time. Hence, the total 
running time of the algorithm is 0(m 2 ), which is 0(n). Therefore, we have 
shown that 2 Color e P. 


6.3 The complexity class NP 

Before we define the class NP, we consider some examples. 

Example 6.3.1 Let G be a graph with vertex set V and edge set E, and 
let k > 1 be an integer. We say that G is k- colorable, if it is possible to give 
each vertex of V a color such that 

1. for each edge (it, v) <G E, the vertices u and v have different colors, and 

2. at most k different colors are used to color all vertices. 

We define the following language: 

kColor := {(G) : the graph G is fc-colorable}. 
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Algorithm 2COLOR 

for i := 1 to m do f(i) := 0; a(i) := 0 endfor; 

/(1) := red; M := 0; 
while M/ra 

do (* Find the minimum index i for which vertex i has not 
been marked, but has a color already *) 
bool := false; i := 1; 
while bool = false and i <nn 

do if a(i) = 0 and f(i) A 0 then bool := true else i := i + 1 endif; 
end while; 

(* If bool = true, then i is the smallest index such that 
a(i ) = 0 and /(*) A 0. 

If bool = false, then for all i, the following holds: if a(i) = 0, then 
f(i) = 0; because M < m, there is at least one such i. *) 
if bool = true 
then for j := 1 to m 
do if E(i,j) = 1 

then if f(i) = f(j) 

then return NO and terminate 
else if f(j) = 0 

then if f{i) = red 

then f(j) := blue 
else f(j) := red 
endif 

endif 

endif 

endif 

endfor; 

a(i) := 1; M := M+ 1; 
else i ;= 1; 

while a(i) ^ 0 do i := i + 1 endwhile; 

(* an unvisited connected component starts at vertex i *) 
f(i) := red 

endif 

endwhile; 

return YES 

Figure 6.2: An algorithm that decides whether or not a graph G is 2- 
colorable. 


We have seen that for k = 2, this problem is in the class P. For k > 3, it 
is not known whether there exists an algorithm that decides, in polynomial 
time, whether or not any given graph is fc-colorable. In other words, for 
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k > 3, it is not known whether or not k Color is in the class P. 

Example 6.3.2 Let G be a graph with vertex set V = {1,2,..., m} and 
edge set E. A Hamilton cycle is a cycle in G that visits each vertex exactly 
once. Formally, it is a sequence iq, v 2 ,..., v m of vertices such that 

1. {ui, v 2 , • • •, v m } = V, and 

2- {(«!, v 2 ), (y 2 , {v m -i, v m ), (y m , ui)} C E. 

We define the following language: 

HC := {(G) : the graph G contains a Hamilton cycle}. 

It is not known whether or not HC is in the class P. 

Example 6.3.3 The sum of subset language is defined as follows: 

SOS {(ai, a 2 ,..., a m , b) : m, Gq, a 2 ,..., a m , b e N 0 and 

31 C {1,2 = b }- 

Also in this case, no polynomial-time algorithm is known that decides the 
language SOS. That is, it is not known whether or not SOS is in the class 

P. 

Example 6.3.4 An integer x > 2 is a prime number, if there are no a, b e N 
such that a ^ x, b ^ x, and x = ah. Hence, the language of all non-primes 
that are greater than or equal to two, is 

NPrim {(x) : x > 2 and x is not a prime number}. 

It is not obvious at all, whether or not NPrim is in the class P. In fact, it 
was shown only in 2002 that NPrim is in the class P. 

Observation 6.3.5 The four languages above have the following in com¬ 
mon: If someone gives us a “solution” for any given input, then we can 
easily, i.e., in polynomial time, verify whether or not this “solution” is a cor¬ 
rect solution. Moreover, for any input to each of these four problems, there 
exists a “solution” whose length is polynomial in the length of the input. 
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Let us again consider the language k Color. Let G be a graph with vertex 
set V = {1, 2,..., m} and edge set E, and let A; be a positive integer. We 
want to decide whether or not G is fc-colorable. A “solution” is a coloring of 
the nodes using at most k different colors. That is, a solution is a sequence 
/i, f 2 , ■ ■ ■ , /m- (Interpret this as: vertex i receives color f tl 1 < i < rri). This 
sequence is a correct solution if and only if 

1. fi E {1,2,, k}, for all i with 1 < i < m, and 

2. for all i with 1 < i < m, and for all j with 1 < j < m, if (i,j) G E, 
then fi ± fj. 

If someone gives us this solution (i.e., the sequence f \, / 2 ,..., f m ), then 
we can verify in polynomial time whether or not these two conditions are 
satisfied. The length of this solution is O(mlogfc): for each i, we need about 
log A: bits to represent fi. Hence, the length of the solution is polynomial in 
the length of the input, i.e., it is polynomial in the number of bits needed to 
represent the graph G and the number k. 

For the Hamilton cycle problem, a solution consists of a sequence v\, 
v 2 ,... ,v m ol vertices. This sequence is a correct solution if and only if 

1. {iq, v 2 , ■ ■ ■, v m } = { 1 , 2 ,..., m} and 

2- {(ui, v 2 ), ( v 2 , v 3 ),..., (u m _i, v m ), ( v m , ui)} C E. 

These two conditions can be verified in polynomial time. Moreover, the 
length of the solution is polynomial in the length of the input graph. 

Consider the sum of subset problem. A solution is a sequence c 3 , c 2 ,..., c m . 
It is a correct solution if and only if 

1. Cj G {0,1}, for all i with 1 < % < m, and 

2. Y1T= i c * a * = b- 

Hence, the set / C {1,2,..., m} in the definition of SOS is the set of indices 
i for which c* = 1. Again, these two conditions can be verified in polynomial 
time, and the length of the solution is polynomial in the length of the input. 

Finally, let us consider the language NPrim. Let x > 2 be an integer. 
The integers a and b form a “solution” for x if and only if 
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1. 2 < a < x, 

2. 2 < b < x, and 

3. x — ab. 

Clearly, these three conditions can be verified in polynomial time. Moreover, 
the length of this solution, i.e., the total number of bits in the binary rep¬ 
resentations of a and b, is polynomial in the number of bits in the binary 
representation of x. 

Languages having the property that the correctness of a proposed “solu¬ 
tion” can be verified in polynomial time, form the class NP: 

Definition 6.3.6 A language A belongs to the class NP, if there exist a 
polynomial p and a language Be P, such that for every string w, 

w E A -<=>- 3s : |s| < p(\w\) and {w, s) E B. 

In words, a language A is in the class NP, if for every string w, w E A if 
and only if the following two conditions are satisfied: 

1. There is a “solution” s, whose length |s| is polynomial in the length of 
w (i.e., |s| < p(\w\), where p is a polynomial). 

2. In polynomial time, we can verify whether or not s is a correct “solu¬ 
tion” for w (i.e., (w, s) E B and B E P). 

Hence, the language B can be regarded to be the “verification language”: 

B = {(w, s) : s is a correct “solution” for w}. 

We have given already informal proofs of the fact that the languages 
kColor, HC, SOS , and NPrim are all contained in the class NP. Below, we 
formally prove that NPrim E NP. To prove this claim, we have to specify 
the polynomial p and the language Be P. First, we observe that 

NPrim = { (x) : there exist a and b in N such that , , 

2 < a < x, 2 <b < x and x — ab}. ^ ' ' 

We define the polynomial p by p(n) := n + 2, and the language B as 

B {(x, a,b) : x > 2,2 < a < x, 2 <b < x and x = ab}. 
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It is obvious that BeP: For any three positive integers x, a, and b, we 
can verify in polynomial time whether or not (x, a , b ) <E B. In order to do 
this, we only have to verify whether or not x>2, 2<a<x,2<b<x, 
and x — ab. If all these four conditions are satisfied, then (x, a, b) G B. If at 
least one of them is not satisfied, then (x, a, b) B. 

It remains to show that for all x G N: 

(x) G NPrim 3a, b : |(a, 6)| < |(x)| + 2 and (x, a, b) G B. (6.2) 

(Remember that | ( x ) | denotes the number of bits in the binary representation 
of x; | (a, 6) | denotes the total number of bits of a and b, i.e., | (a, t>) | — 
|(a)| + |(6)|.) 

Let x G NPrim. It follows from (6.1) that there exist a and b in N, such 
that 2 < a < x, 2 < b < x, and x = ab. Since x = ab > 2 • 2 = 4 > 2, it 
follows that (x, a, b) G B. Hence, it remains to show that 

KM)I < |(®)| +2. 

The binary representation of x contains [log xj+l bits, i.e., |(x)| = [logxj+l. 
We have 


KM) | = 


< 


< 


([log aj + 1) + ([log 6J +1) 

log a + log b + 2 

log ab+2 

log x + 2 

(log xj + 3 

|(x)| + 2. 


This proves one direction of (6.2). 

To prove the other direction, we assume that there are positive integers 
a and b, such that |(M)I < | (a;) | + 2 and ( x,a,b) e B. Then it follows 
immediately from (6.1) and the definition of the language B , that x G NPrim. 
Hence, we have proved the other direction of (6.2). This completes the proof 
of the claim that 

NPrim G NP. 
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6.3.1 P is contained in NP 

Intuitively, it is clear that P C NP, because a language is 

• in P, if for every string w, it is possible to compute the “solution” s in 
polynomial time, 

• in NP, if for every string w and for any given “solution” s, it is possible 
to verify in polynomial time whether or not s is a correct solution for 
w (hence, we do not need to compute the solution s ourselves, we only 
have to verify it). 

We give a formal proof of this: 

Theorem 6.3.7 P C NP. 

Proof. Let A £ P. We will prove that A £ NP. Define the polynomial p 

by p{n) := 0 for all n £ N 0 , and define 

B := {( w , e) : w £ A}. 

Since deP, the language B is also contained in P. It is easy to see that 
w £ A -<=>■ 3s : |s| < p(|«?|) = 0 and (w, s) £ B. 

This completes the proof. ■ 


6.3.2 Deciding NP-languages in exponential time 

Let us look again at the definition of the class NP. Let A be a language in 
this class. Then there exist a polynomial p and a language B £ P, such that 
for all strings w, 

w £ A -<=>■ 3s : |s| < p(|iu|) and (w, s ) £ B. (6.3) 

How do we decide whether or not any given string w belongs to the language 
A? If we can find a string s that satisfies the right-hand side in (6.3), then 
we know that w £ A. On the other hand, if there is no such string s, then 
w 0 A. How much time do we need to decide whether or not such a string s 
exists? 
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Algorithm NonPrime 

(* decides whether or not ( x) £ NPrim *) 

ifx = 0orx = lorx = 2 

then return NO and terminate 

else a := 2; 

while a < x 
do if x mod a = 0 

then return YES and terminate 
else a := a + 1 
endif 
endwhile; 
return NO 
endif 


Figure 6.3: An algorithm that decides whether or not a number x is contained 
in the language NPrim. 


For example, let A be the language NPrim , and let The algorithm 

in Figure 6.3 decides whether or not (x) £ NPrim. 

It is clear that this algorithm is correct. Let n be the length of the binary 
representation of x, i.e., n = [log x\ + 1. If x > 2 and x is a prime number, 
then the while-loop makes x — 2 iterations. Therefore, since n — 1 = [log x\ < 
logx, the running time of this algorithm is at least 

x - 2 > 2 n ~ 1 - 2, 

i.e., it is at least exponential in the length of the input. 

We now prove that every language in NP can be decided in exponential 
time. Let A be an arbitrary language in NP. Let p be the polynomial, and 
let B £ P be the language such that for all strings w, 

w £ A -<=>■ 3s : |s| < p(|rc|) and (w, s) £ B. (6.4) 

The following algorithm decides, for any given string w, whether or not 
w £ A. It does so by looking at all possible strings s for which |s| < p(\w\): 

for all s with |s| < p(|iu|) 
do if {w, s) £ B 
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then return YES and terminate 

endif 

endfor; 

return NO 

The correctness of the algorithm follows from (6.4). What is the running 
time? We assume that w and s are represented as binary strings. Let n be 
the length of the input, i.e., n — |iy|. 

How many binary strings s are there whose length is at most p(\w\)7 Any 
such s can be described by a sequence of length p(\w\) = p{n ), consisting of 
the symbols “0”, “1”, and the blank symbol. Hence, there are at most 3 p(p ? 
many binary strings s with |s| < p{n). Therefore, the for-loop makes at most 
3 p ( n ) iterations. 

Since B e P, there is an algorithm and a polynomial q, such that this 
algorithm, when given any input string z, decides in q(\z\) time, whether or 
not z E B. This input z has the form (w, s), and we have 

\z\ = |iu| + |s| < |iu| +p(\w\) — n +p(n). 

It follows that the total running time of our algorithm that decides whether 
or not vj G A, is bounded from above by 

3 p (”) ■ q(n + p{n)) < 2 2p ^ • q(n + p{n)) 

< 2 2p ( n ) • 2 q ( n+p( - n ^ 

= 2 p ' (n) , 

where p' is the polynomial that is defined by p'(n) := 2 p(n) + q(n+p(n)). 

If we define the class EXP as 

EXP := {A : there exists a polynomial p, such that A can be 
decided in time 2 P ?? } , 
then we have proved the following theorem. 

Theorem 6.3.8 NP C EXP. 

6.3.3 Summary 

• PC NP. It is not known whether P is a proper subclass of NP, or 
whether P = NP. This is one of the most important open problems in 
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computer science. If you can solve this problem, then you will get one 
million dollars; not from us, but from the Clay Mathematics Institute, 
see 


http://www.claymath.org/prizeproblems/index.htm 
Most people believe that P is a proper subclass of NP. 

• NP C EXP, i.e., each language in NP can be decided in exponential 
time. It is not known whether NP is a proper subclass of EXP, or 
whether NP = EXP. 

• It follows from P C NP and NP C EXP, that P C EXP. It can 
be shown that P is a proper subset of EXP, i.e., there exist languages 
that can be decided in exponential time, but that cannot be decided in 
polynomial time. 

• P is the class of those languages that can be decided efficiently , i.e., in 
polynomial time. Sets that are not in P, are not efficiently decidable. 


6.4 Non-deterministic algorithms 

The abbreviation NP stands for Non-deterministic Polynomial time. The al¬ 
gorithms that we have considered so far are deterministic , which means that 
at any time during the computation, the next computation step is uniquely 
determined. In a non-deterministic algorithm, there are one or more possi¬ 
bilities for being the next computation step, and the algorithm chooses one 
of them. 

To give an example, we consider the language SOS, see Example 6.3.3. 
Let m, aq, a 2 , ..., a m , and b be elements of N 0 . Then 

(ai, a 2 , ■ ■ ■, a m , b) G SOS there exist Cp c 2 ,..., Cm G {0,1}, 

such that YliL i °i a i — b- 

The following non-deterministic algorithm decides the language SOS: 

Algorithm SOS(m, ai, a 2 ,..., a m , b): 
s *— 0; 

for i := 1 to m 

do s := s | s := s + a* 
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endfor; 

if s — b 

then return YES 
else return NO 
endif 

The line 

s s | s s + a* 

means that either the instruction u s s” or the instruction “s := s + a” is 
executed. 

Let us assume that (ai, a 2 ,..., a m , b) G SOS. Then there are ci, c 2 ,..., c m G 
{0,1} such that c i a i = b. Assume our algorithm does the following, for 
each i with 1 < i < m: In the i-th iteration, 

• if q = 0, then it executes the instruction “s := s”, 

• if q — 1, then it executes the instruction “s := s + 

Then after the for-loop, we have s = b, and the algorithm returns YES; 
hence, the algorithm has correctly found out that (ai, a 2 ,..., a m , b) e SOS. 

In other words, in this case, there exists at least one accepting computation. 

On the other hand, if (ai, a 2 ,..., a m , b) 0 SOS, then the algorithm always 
returns NO, no matter which of the two instructions is executed in each 
iteration of the for-loop. In this case, there is no accepting computation. 

Definition 6.4.1 Let M be a non-deterministic algorithm. We say that M 
accepts a string w, if there exists at least one computation that, on input w, 
returns YES. 

Definition 6.4.2 We say that a non-deterministic algorithm M decides a 
language A in time T , if for every string w, the following holds: vj <G A if 
and only if there exists at least one computation that, on input w, returns 
YES and that takes at most T(\w\) time. 

The non-deterministic algorithm that we have seen above decides the 
language SOS in linear time: Let (ai, a 2 ,..., a m , b) e SOS, and let n be the 
length of this input. Then 

n = l(°i)l + I(a 2 )| + • • • + |(a m )| + |(6)| > m. 
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For this input, there is a computation that returns YES and that takes 
0{m ) = 0{n) time. 

As in Section 6.2, we define the notion of “polynomial time” for non- 
deterministic algorithms. The following theorem relates this notion to the 
class NP that we defined in Definition 6.3.6. 

Theorem 6.4.3 A language A is in the class NP if and only if there exists 
a non-deterministic Turing machine (or Java program) that decides A in 
polynomial time. 


6.5 NP-complete languages 

Languages in the class P are considered easy , i.e., they can be decided in 
polynomial time. People believe (but cannot prove) that P is a proper sub¬ 
class of NP. If this is true, then there are languages in NP that are hard, 
i.e., cannot be decided in polynomial time. 

Intuition tells us that if P ^ NP, then the hardest languages in NP are 
not contained in P. These languages are called NP- complete. In this section, 
we will give a formal definition of this concept. 

If we want to talk about the “hardest” languages in NP, then we have to 
be able to compare two languages according to their “difficulty”. The idea is 
as follows: We say that a language B is “at least as hard” as a language A, 
if the following holds: If B can be decided in polynomial time, then A can 
also be decided in polynomial time. 

Definition 6.5.1 Let A C {0,1}* and B C {0,1}* be languages. We say 
that A <p B, if there exists a function 

/: {0,1}- ->{0,1K 


such that 

1. / G FP and 

2. for all strings w in {0,1}*, 

w G A <*=£> f(w) G B. 
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If A < P B, then we also say that “B is at least as hard as A ”, or “A is 
polynomial-time reducible to B". 

We first show that this formal definition is in accordance with the intuitive 
definition given above. 

Theorem 6.5.2 Let A and B be languages such that B e P and A <p B. 
Then A e P. 

Proof. Let / : {0, 1}* —> {0, 1}* be the function in FP for which 

we A «=► f(w) e B. (6.5) 

The following algorithm decides whether or not any given binary string w is 
in A: 


u := f(w)] 
if u e B 

then return YES 
else return NO 
endif 

The correctness of this algorithm follows immediately from (6.5). So it 
remains to show that the running time is polynomial in the length of the 
input string w. 

Since / e FP, there exists a polynomial p such that the function / can 
be computed in time p. Similarly, since Be P, there exists a polynomial q, 
such that the language B can be decided in time q. 

Let n be the length of the input string w, i.e., n — \w\. Then the length 
of the string u is less than or equal to p(\vj\) — p(n). (Why?) Therefore, the 
running time of our algorithm is bounded from above by 

KM) + KM) < p{n) + q(p{n)). 

Since the function p', defined by p'(n) := p(n) +q(p(n)), is a polynomial, this 
proves that iGP. ■ 

The following theorem states that the relation <p is reflexive and tran¬ 
sitive. We leave the proof as an exercise. 

Theorem 6.5.3 Let A, B, and C be languages. Then 
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1. A < P A, and 

2. if A < P B and B < P C, then A < P C. 

We next show that the languages in P are the easiest languages in NP: 

Theorem 6.5.4 Let A be a language in P, and let B be an arbitrary lan¬ 
guage such that B ^ 0 and B / {0,1}*. Then A < P B. 

Proof. We choose two strings u and v in {0,1}*, such that u e B and v 0 B. 
(Observe that this is possible.) Define the function / : {0,1}* —>• {0,1}* by 

\ f u if w G A, 

f( w ) ■— | v if w ^ A. 

Then it is clear that for any binary string w, 

w G A ■<=>■ f(w) e B. 

Since A 6 P, the function / can be computed in polynomial time, i.e., 
/ e FP. ■ 

6.5.1 Two examples of reductions 

Sum of subsets and knapsacks 

We start with a simple reduction. Consider the two languages 

SOS := {(ai, • • ■, a m , b) : m, ai,..., a m , b e N 0 and there exist 

Ci,..., c m G {0,1}, such that Yl'iLi c i a i ~ b} 

and 

KS := {{wu...,w mi ki,...,km,W,K): 

m, wi ,..., w m , ki ,..., k m , W,KeNo 

and there exist Ci,,.., c m € {0,1}, 

such that YliL i c i w i < W and Y1T=i c ^i > K}. 

The notation KS stands for knapsack: We have m pieces of food. The 
i-th piece has weight Wi and contains ki calories. We want to decide whether 
or not we can fill our knapsack with a subset of the pieces of food such that 
the total weight is at most W, and the total amount of calories is at least K. 
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Theorem 6.5.5 SOS < P KS. 

Proof. Let us first see what we have to show. According to Definition 6.5.1, 
we need a function / £ FP, that maps input strings for SOS to input strings 
for KS, in such a way that 

(ai,..., a m , b) £ SOS /((ai,..., a m , b)) £ KS. 

In order for f((ai,...,a m ,b)) to be an input string for KS, this function 
value has to be of the form 

/((ai,..., a m , b)) = (iui,..., k u ..., k m , W, K). 

We define 

/((ai,..., a m , 6)) := (ai,..., a m , Oi,..., a m , b, b). 

It is clear that / £ FP. We have 
(ai,..., a m , b) £ SOS 

-<=>■ there exist c\,..., Cm £ {0,1} such that YLT= i c * a * = ^ 

there exist ci,..., c m £ {0,1} such that Y^i=i c * a * — ^ an( ^ i 
(ai,..., a m , oi,..., a m , 5,6) £ A5 
^ f{(a 1 ,...,a m ,b))eKS. 


Cliques and Boolean formulas 

We will define two languages A = 3 SAT and B = Clique that have, at 
first sight, nothing to do with each other. Then we show that, nevertheless, 
A B. 

Let G be a graph with vertex set V and edge set A 1 . A subset V of V is 
called a clique, if each pair of distinct vertices in V' is connected by an edge 
in E. We define the following language: 

Clique := {(G, k) : k £ N and G has a clique with k vertices}. 

We encourage you to prove the following claim: 
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Theorem 6.5.6 Clique 6 NP. 

Next we consider Boolean formulas p, with variables x \, x 2 ,..., x m , hav¬ 
ing the form 

p = C 1 A C 2 A ... A C k , ( 6 . 6 ) 

where each (7j, 1 < i < k, is of the form 

Ci = 4 V 4 V 4 . 

Each 4 is either a variable or the negation of a variable. An example of such 
a formula is 

p — (xi V ->xi V ~ix 2 ) A (x 3 V x 2 V x 4 ) A (->Xi V -ix 3 V - 1 X 4 ). 

A formula p of the form ( 6 . 6 ) is said to be satisfiable, if there exists a truth- 
value in { 0 , 1 } for each of the variables xi, x 2 ,..., x m , such that the entire 
formula p is true. Our example formula is satisfiable: If we take X\ — 0 and 
x 2 = 1 , and give x 3 and x 4 an arbitrary value, then 

p — (0 V 1 V 0) A (x 3 V 1 V x 4 ) A (1 V - 1 X 3 V - 1 X 4 ) = 1. 

We define the following language: 

3 SAT := {( 97 ) : p is of the form ( 6 . 6 ) and is satisfiable}. 

Again, we encourage you to prove the following claim: 

Theorem 6.5.7 3SAT € NP. 

Observe that the elements of Clique (which are pairs consisting of a graph 
and a positive integer) are completely different from the elements of 3SAT 
(which are Boolean formulas). We will show that 3SAT <p Clique. Recall 
that this means the following: If the language Clique can be decided in 
polynomial time, then the language 3SAT can also be decided in polynomial 
time. In other words, any polynomial-time algorithm that decides Clique can 
be converted to a polynomial-time algorithm that decides 3 SAT. 

Theorem 6.5.8 3 SAT < P Clique. 


Source: www.csitsansar.com 



212 


Chapter 6. Complexity Theory 


Proof. We have to show that there exists a function / e FP, that maps 
input strings for 3 SAT to input strings for Clique , such that for each Boolean 
formula (p that is of the form (6.6), 

(< p) G 3SAT f((<p)) G Clique. 

The function / maps the binary string encoding an arbitrary Boolean formula 
tp to a binary string encoding a pair (G, k), where G is a graph and k is a 
positive integer. We have to define this function / in such a way that 

(p is satisfiable -<=>- G has a clique with k vertices. 


Let 

ip — C\ A G 2 A ... A Cfc 

be an arbitrary Boolean formula in the variables x%,x 2 ,, x m , where each 
Ci, 1 < i < fc, is of the form 

Ci = 4 v 4 v 4. 

Remember that each t a is either a variable or the negation of a variable. 

The formula (p is mapped to the pair (G, k), where the vertex set V and 
the edge set E of the graph G are defined as follows: 

• V — (4, 4,..., 4, v$, 4}. The idea is that each vertex v l a corre¬ 

sponds to one term C a . 

• The pair (u®, i’l) of vertices form an edge in E if and only if 

— i ^ j and 

— t\ is not the negation of E b . 

To give an example, let ip be the Boolean formula 

ip — (x i V ~>x 2 V ~>x 3 ) A (-ixi V x 2 V x 3 ) A (aq Va 2 V x 3 ), (6.7) 

i.e., k — 3, Ci — xi V ~>x 2 V ~>x 3 , C 2 — ~>xi Vr 2 V x 3 , and C 3 — xi V x 2 V x 3 . 
The graph G that corresponds to this formula is given in Figure 6.4. 

It is not difficult to see that the function / can be computed in polynomial 
time. So it remains to prove that 

ip is satisfiable G has a clique with k vertices. (6.8) 


Source: www.csitsansar.com 




Figure 6.4: The formula ip in (6.7) is mapped to this graph. The vertices on 
the top represent G\: the vertices on the left represent C 2 ; the vertices on 
the right represent C 3 . 


To prove this, we first assume that the formula 


(p — Ci A C 2 A ... A C k 


is satisfiable. Then there exists a truth-value in {0,1} for each of the variables 
Xi,x%i, x m , such that the entire formula p is true. Hence, for each i with 
1 < i < k, there is at least one term t 0 in 

q = 4 v 4 v 4 

that is true (i.e., has value 1). 

Let V' be the set of vertices obtained by choosing for each i, 1 < i < k, 
exactly one vertex u* such that t a has value 1. 

It is clear that V' contains exactly k vertices. We claim that this set is 
a clique in G. To prove this claim, let v l a and v b be two distinct vertices in 
V'. It follows from the definition of V' that i 4 j and i l a — t\ = 1. Hence, 
4 is not the negation of & h . But this means that the vertices v l a and v 3 b are 
connected by an edge in G. 

This proves one direction of (6.8). To prove the other direction, we assume 
that the graph G contains a clique V' with k vertices. 
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The vertices of G consist of k groups, where each group contains exactly 
three vertices. Since vertices within the same group are not connected by 
edges, the clique V' contains exactly one vertex from each group. Hence, for 
each i with 1 < i < k, there is exactly one a, such that v l a e V'. Consider 
the corresponding term t l a . We know that this term is either a variable or 
the negation of a variable, i.e., t a is either of the form Xj or of the form -i Xj. 
If t a — Xj, then we give x 3 the truth-value 1. Otherwise, we have t a = -i Xj, 
in which case we give Xj the truth-value 0. Since V' is a clique, each variable 
gets at most one truth-value. If a variable has no truth-value yet, then we 
give it an arbitrary truth-value. 

If we substitute these truth-values into <p, then the entire formula has 
value 1. Hence, <p is satisfiable. ■ 

In order to get a better understanding of this proof, you should verify the 
proof for the formula p in (6.7) and the graph G in Figure 6.4. 

6.5.2 Definition of NP-completeness 

Reductions, as defined in Definition 6.5.1, allow us to compare two language 
according to their difficulty. A language B in NP is called NP-complete, 
if B belongs to the most difficult languages in NP; in other words, B is at 
least as hard as any other language in NP. 

Definition 6.5.9 Let B C {0,1}* be a language. We say that B is NP- 
complete, if 

1 .Be NP and 

2. A < P B, for every language A in NP. 

Theorem 6.5.10 Let B be an NP -complete language. Then 

Be p P = NP. 

Proof. Intuitively, this theorem should be true: If the language B is in P, 
then B is an easy language. On the other hand, since B is NP-complete, 
it belongs to the most difficult languages in NP. Hence, the most difficult 
language in NP is easy. But then all languages in NP must be easy, i.e., 

P = NP. 
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We give a formal proof. Let us first assume that B e P. We already 
know that P C NP. Hence, it remains to show that NP C P. Let A be an 
arbitrary language in NP. Since B is NP-complete, we have A < P B. Then, 
by Theorem 6.5.2, we have A e P. 

To prove the converse, assume that P = NP. Since B 6 NP, it follows 
immediately that B 6 P. ■ 


Theorem 6.5.11 Let B and C be languages, such that C e NP and B < P 
C. If B is NP -complete, then C is also NP -complete. 

Proof. First, we give an intuitive explanation of the claim: By assumption, 
B belongs to the most difficult languages in NP, and C is at least as hard as 
B. Since C e NP, it follows that C belongs to the most difficult languages 
in NP. Hence, C is NP-complete. 

To give a formal proof, we have to show that A < P C, for all languages A 
in NP. Let A be an arbitrary language in NP. Since B is NP-complete, we 
have A < P B. Since B < P C, it follows from Theorem 6.5.3, that A < P C. 
Therefore, C is NP-complete. ■ 

Theorem 6.5.11 can be used to prove the NP-completeness of languages: 
Let C be a language, and assume that we want to prove that C is NP- 
complete. We can do this in the following way: 

1. We first prove that C e NP. 

2. Then we find a language B that looks “similar” to C , and for which 
we already know that it is NP-complete. 

3. Finally, we prove that B < P C. 

4. Then, Theorem 6.5.11 tells us that C is NP-complete. 

Of course, this leads to the question “How do we know that the language 
B is NP-complete?” In order to apply Theorem 6.5.11, we need a “first” NP- 
complete language; the NP-completeness of this language must be proven 
using Definition 6.5.9. 

Observe that it is not clear at all that there exist NP-complete languages! 
For example, consider the language 3 SAT. If we want to use Definition 6.5.9 
to show that this language is NP-complete, then we have to show that 
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• 3 SAT e NP. We know from Theorem 6.5.7 that this is true. 

• A <p 3SAT, for every language A 6 NP. Hence, we have to show this 
for languages A such as kColor, HC, SOS, NPrim, KS, Clique, and 
for infinitely many other languages. 

In 1971, Cook has exactly done this: He showed that the language 3SAT 
is NP-complete. Since his proof is rather technical, we will prove the NP- 
completeness of another language. 

6.5.3 An NP-complete domino game 

We are given a finite collection of tile types. For each such type, there are 
arbitrarily many tiles of this type. A tile is a square that is partitioned into 
four triangles. Each of these triangles contains a symbol that belongs to a 
finite alphabet E. Hence, a tile looks as follows: 



We are also given a square frame, consisting of cells. Each cell has the same 
size as a tile, and contains a symbol of E. 

The problem is to decide whether or not this domino game has a solution. 
That is, can we completely fill the frame with tiles such that 

• for any two neighboring tiles s and s', the two triangles of s and s' that 
touch each other contain the same symbol, and 

• each triangle that touches the frame contains the same symbol as the 
cell of the frame that is touched by this triangle. 

There is one final restriction: The orientation of the tiles is fixed, they cannot 
be rotated. 

Let us give a formal definition of this problem. We assume that the sym¬ 
bols belong to the finite alphabet E = {0, l} m , i.e., each symbol is encoded 
as a bit-string of length m. Then, a tile type can be encoded as a tuple of 
four bit-strings, i.e., as an element of E 4 . A frame consisting of t rows and t 
columns can be encoded as a string in E 4t . 
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We denote the language of all solvable domino games by Domino: 

Domino := {(m, k,t, R,Ti,... ,T k ) : 

m > 1, k > 1, t > 1, R E E 4t , Ti E £ 4 ,1 < i < k, 
frame R can be filled using tiles of types 

T lt ...,T k .} 

We will prove the following theorem. 

Theorem 6.5.12 The language Domino is NP -complete. 

Proof. It is clear that Domino E NP: A solution consists of a t x t matrix, 
in which the (i, j)-entry indicates the type of the tile that occupies position 
(i, j) in the frame. The number of bits needed to specify such a solution is 
polynomial in the length of the input. Moreover, we can verify in polynomial 
time whether or not any given “solution” is correct. 

It remains to show that 

A <p Domino, for every language A in NP. 

Let A be an arbitrary language in NP . Then there exist a polynomial p and 
a non-deterministic Turing machine M , that decides the language A in time 
p. We may assume that this Turing machine has only one tape. 

On input w — a \02 ■■■ a n , the Turing machine M starts in the start state 
zo, with its tape head on the cell containing the symbol a\. We may assume 
that during the entire computation, the tape head never moves to the left of 
this initial cell. Hence, the entire computation “takes place” in and to the 
right of the initial cell. We know that 

w E A <==? on input w, there exists an accepting computation 
that makes at most p(n) computation steps. 

At the end of such an accepting computation, the tape only contains the 
symbol 1, which we may assume to be in the initial cell, and M is in the final 
state Z\. In this case, we may assume that the accepting computation makes 
exactly p{n) computation steps. (If this is not the case, then we extend the 
computation using the instruction Z\1 —>• Z\1N.) 

We need one more technical detail: We may assume that za —> z'bR and 
za' —> z"h'L are not both instructions of M. Hence, the state of the Turing 
machine uniquely determines the direction in which the tape head moves. 
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We have to define a domino game, that depends on the input string w 
and the Turing machine M, such that 

w G A <==? this domino game is solvable. 

The idea is to encode an accepting computation of the Turing machine M as 
a solution of the domino game. In order to do this, we use a frame in which 
each row corresponds to one computation step. This frame consists of p(n) 
rows. Since an accepting computation makes exactly p{n) computation steps, 
and since the tape head never moves to the left of the initial cell, this tape 
head can visit only p{n) cells. Therefore, our frame will have p(n) columns. 

The domino game will use the following tile types: 

1. For each symbol a in the alphabet of the Turing machine M : 



Intuition: Before and after the computation step, the tape head is not 
on this cell. 

2. For each instruction za —>• z'bR of the Turing machine M: 



Intuition: Before the computation step, the tape head is on this cell; 
the tape head makes one step to the right. 

3. For each instruction za —>• z'bL of the Turing machine M: 
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Intuition: Before the computation step, the tape head is on this cell; 
the tape head makes one step to the left. 

4. For each instruction za —>• z'bN of the Turing machine M: 



Intuition: Before and after the computation step, the tape head is on 
this cell. 

5. For each state z and for each symbol a in the alphabet of the Turing 
machine M, there are two tile types: 



Intuition: The leftmost tile indicates that the tape head enters this cell 
from the left; the righmost tile indicates that the tape head enters this 
cell from the right. 

This specifies all tile types. The p{n) X p(n) frame is given in Figure 6.5. 
The top row corresponds to the start of the computation, whereas the bottom 
row corresponds to the end of the computation. The left and right columns 
correspond to the part of the tape in which the tape head can move. 

The encodings of these tile types and the frame can be computed in 
polynomial time. 

It can be shown that, for any input string w, any accepting computation 
of length p{n) of the Turing machine M can be encoded as a solution of 
this domino game. Conversely, any solution of this domino game can be 
“translated” to an accepting computation of length p{n) of M, on input 
string w. Hence, the following holds. 

w e A <==? there exists an accepting computation that makes 
p(n) computation steps 
the domino game is solvable. 


Source: www.csitsansar.com 






220 


Chapter 6. Complexity Theory 


p(n) 



Therefore, we have A <p Domino. Hence, the language Domino is NP- 
complete. ■ 


An example of a domino game 

We have defined the domino game corresponding to a Turing machine that 
solves a decision problem. Of course, we can also do this for Turing machines 
that compute functions. In this section, we will exactly do this for a Turing 
machine that computes the successor function x —> x + 1. 

We will design a Turing machine with one tape, that gets as input the 
binary representation of a natural number x , and that computes the binary 
representation of x + 1. 

Start of the computation: The tape contains a 0 followed by the binary 
representation of the integer a: 6 Nq. The tape head is on the leftmost bit 
(which is 0), and the Turing machine is in the start state Zq. Here is an 
example, where x = 431: 
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End of the computation: The tape contains the binary representation of 
the number x + 1. The tape head is on the rightmost 1, and the Turing 
machine is in the final state z\. For our example, the tape looks as follows: 


|o|i|i|o|i|i|o|o|o|o|q| 


Our Turing machine will use the following states: 

z 0 : start state; tape head moves to the right 
zi : final state 

z 2 : tape head moves to the left; on its way to the left, it has not read 0 

The Turing machine has the following instructions: 

zqO —y z 0 OR z 2 1 — y z 2 0L 

zqI — y z 0 lR z 2 0 — y Z\1N 

ZoO —)y z 2 OL 

In Figure 6.6, you can see the sequence of states and tape contents of this 
Turing machine on input x = 11. 

We now construct the domino game that corresponds to the computation 
of this Turing machine on input x = 11. Following the general construction 
in Section 6.5.3, we obtain the following tile types: 

1. The three symbols of the alphabet yield three tile types: 



2. The five instructions of the Turing machine yield five tile types: 
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(zo,0) 1 0 1 1 □ 

0 (z 0 , 1) 0 1 1 □ 

0 1 (z 0 ,0) 1 1 □ 

0 1 0 (zo, 1 ) 1 □ 

0 10 1 (z 0 ,1) □ 

0 1 0 1 1 (2b, □) 

0 10 1 (z 2 , 1) □ 

0 1 0 (z 2 , 1) 0 □ 

0 1 (2 2 ,0) 0 0 □ 

0 1 (z lf l) 0 0 □ 

Figure 6.6: The computation of the Turing machine on input x = 11. The 
pair (state,symbol) indicates the position of the tape head. 



3. The states z 0 and z 2 , and the three symbols of the alphabet yield twelve 
tile types: 


\ 0 / 

/fco,o)\ 


zAA# 


\ □ / 
»y# 

/ba.oK 


\ 0 / 
AA# 
A z2 ’A\ 


\ i a 
zA/ # 
A z2 ’T\ 


\ :Q / 

22 y/# 
/b2,oK 







\ 0 / 
#V*o 

A z o,o)\ 


#V*° 


Am, ol\ 


\ 0 A 
AA 2 

A z2 ’A\ 


\ i / 
#\(Z2 

X z2 TK 


\ Q y/ 

AX* 

Az2,A\ 


The computation of the Turing machine on input x = 11 consists of nine 
computation steps. During this computation, the tape head visits exactly 
six cells. Therefore, the frame for the domino game has nine rows and six 
columns. This frame is given in Figure 6.7. In Figure 6.8, you find the 
solution of the domino game. Observe that this solution is nothing but 
an equivalent way of writing the computation of Figure 6.6. Hence, the 
computation of the Turing machine corresponds to a solution of the domino 
game; in fact, the converse also holds. 
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6.5.4 Examples of NP-complete languages 

In Section 6.5.3, we have shown that Domino is NP-complete. Using this 
result, we will apply Theorem 6.5.11 to prove the NP-completeness of some 
other languages. 


Satisfiability 

We consider Boolean formulas <p, in the variables x i, x 2 ,..., x m , having the 
form 

ip = Ci A C 2 A ... A C k , (6.9) 

where each C i} 1 < i < k, is of the following form: 

q = 4 v4v... v4.. 

Each £) is either a variable or the negation of a variable. Such a formula tp 
is said to be satisfiable, if there exists a truth-value in { 0 , 1 } for each of the 
variables xi, x 2 , ■ ■., x m , such that the entire formula ip is true. We define the 
following language: 

SAT := {( 97 ) : ip is of the form (6.9) and is satisfiable}. 

We will prove that SAT is NP-complete. 

It is clear that SAT £ NP. If we can show that 


Domino <p SAT, 


then it follows from Theorem 6.5.11 that SAT is NP-complete. (In Theo¬ 
rem 6.5.11, take B := Domino and C := SAT.) 

Hence, we need a function / £ FP, that maps input strings for Domino 
to input strings for SAT, in such a way that for every domino game D, the 
following holds: 


domino game D is solvable -<=>■ 


the formula encoded by the 
string f((D)) is satisfiable. 


( 6 . 10 ) 


Let us consider an arbitrary domino game D. Let k be the number of 
tile types, and let the frame have t rows and t columns. We denote the tile 
types by 1\, T 2 ,..., T k . 
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We map this domino game D to a Boolean formula </?, such that (6.10) 
holds. The formula ip will have variables 

a*#, 1 < i < t, 1 < j < t, 1 < £ < k. 

These variables can be interpretated as follows: 

xiji — 1 there is a tile of type T e at position (i,j) of the frame. 

We define: 

• For all i and j with 1 < i < t and 1 < j < t: 

G\j := x i:jl V x ij2 V ... V x ijk . 

This formula expresses the condition that there is at least one tile at 
position 

• For all i, j , £ and £' with 1 < i < t ,1 < j < t, and 1 <£<£'< k: 

CijM! ■ V 

This formula expresses the condition that there is at most one tile at 
position 

• For all i, j, t and £! with l<i<t,l<j<t,l<£<k and 1 < £' < k, 
such that i < t and the right symbol on a tile of type T ( is not equal 
to the left symbol on a tile of type T?: 

GijU’ V 

This formula expresses the condition that neighboring tiles in the same 
row “fit” together. There are symmetric formulas for neighboring tiles 
in the same column. 

• For all j and £ with 1 < j <t and 1 < £ < k, such that the top symbol 
on a tile of type T (: is not equal to the symbol at position j of the upper 
boundary of the frame: 

Gji _ia; i ji- 

This formula expresses the condition that tiles that touch the upper 
boundary of the frame “fit” there. There are symmetric formulas for 
the lower, left, and right boundaries of the frame. 
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The formula p is the conjunction of all these formulas CT , C? jU ,, C‘f ]U , , and 
Cj t The complete formula p consists of 

0(t 2 k + t 2 k 2 + t 2 k 2 + tk) = 0(t 2 k 2 ) 

terms, i.e., its length is polynomial in the length of the domino game. This 
implies that <p can be constructed in polynomial time. Hence, the function 
/ that maps the domino game D to the Boolean formula p, is in the class 
FP. It is not difficult to see that (6.10) holds for this function /. Therefore, 
we have proved the following result. 

Theorem 6.5.13 The language SAT is NP -complete. 

In Section 6.5.1, we have defined the language 3SAT. 

Theorem 6.5.14 The language 3SAT is NP-complete. 

Proof. It is clear that 3SAT 6 NP. If we can show that 

SAT < P 3SAT, 

then the claim follows from Theorem 6.5.11. Let 
p — C\ A C 2 A ... A Ck 

be an input for SAT, in the variables x\ , x 2 ,..., x m . We map <p, in polynomial 
time, to an input p' for 3 SAT, such that 

<p is satisfiable -<=>- p' is satisfiable. (6-11) 

For each % with 1 < i < k, we do the following. Consider 

Q = 4 V f 2 V ... V e ki . 

• If ki = 1, then we define 

C[ := 4 V 4 V 4 . 

• If ki — 2, then we define 

C[ 4 V 4 V 4 . 
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• If ki — 3, then we define 

C[ := Q. 


• If ki > 4, then we define 

CJ := (4 V 4 v 4) a (-.4 V 4 v 4) A (--4 v 4 v 4) A ... 
AK- 3 V 4 - 1 V 4 ), 

where zj,..., z l k ._ 3 are new variables. 

Let 

<p' C[ A C' 2 A ... A C{,. 

Then ip' is an input for 3 SAT, and (6.11) holds. ■ 

Theorems 6.5.6, 6.5.8, 6.5.11, and 6.5.14 imply: 

Theorem 6.5.15 The language Clique is NP -complete. 

The traveling salesperson problem 

We are given two positive integers k and m, a set of m cities, and an integer 
m X m matrix M, where 

M(i,j) = the cost of driving from city i to city j, 

for all i, j G {1,2 ,,m}. We want to decide whether or not there is a tour 
through all cities whose total cost is less than or equal to k. This problem is 
NP-complete. 


Bin packing 

We are given three positive integers m, k, and £, a set of rn objects having 
volumes Gq, a 2 ,..., a m , and k bins. Each bin has volume £. We want to 
decide whether or not all objects fit within these bins. This problem is NP- 
complete. 

Here is another interpretation of this problem: We are given m jobs that 
need time Gq, a 2 ,..., a m to complete. We are also given k processors, and an 
integer t. We want to decide whether or not it is possible to divide the jobs 
over the k processors, such that no processor needs more than £ time. 
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Time tables 

We are given a set of courses, class rooms, and professors. We want to 
decide whether or not there exists a time table such that all courses are 
being taught, no two courses are taught at the same time in the same class 
room, no professor teaches two courses at the same time, and conditions such 
as “Prof. L. Azy does not teach before 1pm” are satisfied. This problem is 
NP-complete. 

Motion planning 

We are given two positive integers k and t, a set of k polyhedra, and two 
points s and t in Q 3 . We want to decide whether or not there exists a path 
between s and t, that does not intersect any of the polyhedra, and whose 
length is less than or equal to i. This problem is NP-complete. 

Map labeling 

We are given a map with m cities, where each city is represented by a point. 
For each city, we are given a rectangle that is large enough to contain the 
name of the city. We want to decide whether or not these rectangles can be 
placed on the map, such that 

• no two rectangles overlap, 

• For each i with 1 < i < m, the point that represents city i is a corner 
of its rectangle. 

This problem is NP-complete. 

This list of NP-complete problems can be extended almost arbitrarily: 
For thousands of problems, it is known that they are NP-complete. For all 
of these, it is not known, whether or not they can be solved efficiently (i.e., 
in polynomial time). Collections of NP-complete problems can be found in 
the book 

• M.R. Garey and D.S. Johnson. Computers and Intractability: A Guide 
to the Theory of NY*-Completeness. W.H. Freeman, New York, 1979, 

and on the web page 

http://www.nada.kth.se/~viggo/wwwcompendium/ 


Source: www.csitsansar.com 



230 


Chapter 6. Complexity Theory 


Exercises 

6.1 Prove that the function F : N —> N, defined by F(x) := 2 X , is not in FP. 

6.2 Prove Theorem 6.5.3. 

6.3 Prove that the language Clique is in the class NP. 

6.4 Prove that the language 3SAT is in the class NP. 

6.5 We define the following languages: 

• Sum of subset: 

SOS := {(ai, o 2 ,.. ■, a m , b) : 3/ C {1,2,..., m}, a* = b}. 

iei 


• Set partition: 

SP : = {(ai, a 2 , ■ ■ ■ , a m ) : 31 C {1, 2,..., m}, ^ a* = ^ a*}. 

i&i igi 

• Bin packing: BP is the set of all strings (si, s 2) • • •, s m , B) for which 

1 . 0 < Si < 1, for all i, 

2. B eN, 

3. the numbers si, s 2 ,..., s m fit into B bins, where each bin has size 
one, i.e., there exists a partition of {1,2 ,..., m} into subsets I k , 
1 <k < B, such that Eie/ fc s i < 1 f° r all k, 1 < k < B. 

For example, (1/6,1/2,1/5,1/9, 3/5,1/5,1/2,11/18, 3) e BP , because 
the eight fractions fit into three bins: 

1/6 + 1/9 + 11/18 < 1, 1/2 + 1/2 = 1, and 1/5 + 3/5 + 1/5 = 1. 

1 . Prove that SOS <p SP. 

2 . Prove that the language SOS is NP-complete. You may use the fact 
that the language SP is NP-complete. 
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3. Prove that the language BP is NP-complete. Again, you may use the 
fact that the language SP is NP-complete. 

6.6 Prove that 3 Color <p 3SAT. 

Hint: For each vertex i, and for each of the three colors k, introduce a 
Boolean variable x^. 

6.7 The (0,1 )-integer programming language IP is defined as follows: 

IP := {(A, c) : A is an integer m X n matrix for some m,n 6 N, 
c is an integer vector of length m, and 
3x G {0, l} n such that Ax < c (componentwise) }. 
Prove that the language IP is NP-complete. You may use the fact that 
the language SOS is NP-complete. 

6.8 Let ip be a Boolean formula in the variables x 1} x 2 , ■.., x m . 

We say that <p is in disjunctive normal form (DNF) if it is of the form 

(p = C 1 VC 2 V... VC k , (6.12) 

where each (7$, 1 < * < k, is of the following form: 

c i = e 1 a 4 a... a 4 . 

Each R is a literal, which is either a variable or the negation of a variable. 
We say that p is in conjunctive normal form (CNF) if it is of the form 

p = C x AC 2 A... AC k , (6.13) 

where each Ci, 1 < i < k, is of the following form: 

q = 4 v 4v...v4.. 

Again, each t- is a literal. 

We define the following two languages: 

DNFSAT :={(<^) : <p is in DNF-form and is satisfiable}, 

and 

CNFSAT := {(<£>) : p is in CNF-form and is satisfiable}. 
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1. Prove that the language DNFSAT is in P. 

2 . What is wrong with the following argument: Since we can rewrite 
any Boolean formula in DNF-form, we have CNFSAT < P DNFSAT. 
Hence, since CNFSAT is NP-complete and since DNFSAT G P, we 
have P = NP. 

3. Prove directly that for every language A in P, A <p CNFSAT. “Di¬ 
rectly” means that you should not use the fact that CNFSAT is NP- 
complete. 

6.9 Prove that the polynomial upper bound on the length of the string y in 
the definition of NP is necessary, in the sense that if it is left out, then any 
decidable language would satisfy the condition. 

More precisely, we say that the language A belongs to the class D, if there 
exists a language BgP, such that for every string w, 

w G A •<=>- 3y : (w, y) G B. 

Prove that D is equal to the class of all decidable languages. 
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We have seen several different models for “processing” languages, i.e., pro¬ 
cessing sets of strings over some finite alphabet. For each of these models, 
we have asked the question which types of languages can be processed, and 
which types of languages cannot be processed. In this final chapter, we give 
a brief summary of these results. 

Regular languages: This class of languages was considered in Chapter 2. 
The following statements are equivalent: 

1 . The language A is regular, i.e., there exists a deterministic finite au¬ 
tomaton that accepts A. 

2 . There exists a nondeterministic finite automaton that accepts A. 

3. There exists a regular expression that describes A. 

This claim was proved by the following conversions: 

1. Every nondeterministic finite automaton can be converted to an equiv¬ 
alent deterministic finite automaton. 

2. Every deterministic finite automaton can be converted to an equivalent 
regular expression. 

3. Every regular expression can be converted to an equivalent nondeter¬ 
ministic finite automaton. 
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We have seen that the class of regular languages is closed under the regular 
operations: If A and B are regular languages, then 

1. A U B is regular, 

2. AB is regular, 

3. A* is regular, 

4. A is regular, and 

5. A fi B is regular. 

Finally, the pumping lemma for regular languages gives a property that 
every regular language possesses. We have used this to prove that languages 
such as {a n b n : n > 0} are not regular. 

Context-free languages: This class of languages was considered in Chap¬ 
ter 3. We have seen that every regular language is context-free. Moreover, 
there exist languages, for example { a n b n : n > 0}, that are context-free, but 
not regular. The following statements are equivalent: 

1. The language A is context-free, i.e., there exists a context-free grammar 
whose language is A. 

2. There exists a context-free grammar in Chomsky normal form whose 
language is A. 

3. There exists a nondeterministic pushdown automaton that accepts A. 
This claim was proved by the following conversions: 

1. Every context-free grammar can be converted to an equivalent context- 
free grammar in Chomsky normal form. 

2. Every context-free grammar in Chomsky normal form can be converted 
to an equivalent nondeterministic pushdown automaton. 

3. Every nondeterministic pushdown automaton can be converted to an 
equivalent context-free grammar. (This conversion was not covered in 
this book.) 
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Nondeterministic pushdown automata are more powerful than determin¬ 
istic pushdown automata: There exists a nondeterministic pushdown au¬ 
tomaton that accepts the language 

{vbw : v e {a, b}*, w 6 {a, b}*, |u| = |w|}, 

but there is no deterministic pushdown automaton that accepts this language. 
(We did not prove this in this book.) 

We have seen that the class of context-free languages is closed under 
the union, concatenation, and star operations: If A and B are context-free 
languages, then 

1. A U B is context-free, 

2. AB is context-free, and 

3. A* is context-free. 

However, 

1 . the intersection of two context-free languages is not necessarily context- 
free, and 

2 . the complement of a context-free language is not necessarily context- 
free. 

Finally, the pumping lemma for context-free languages gives a property 
that every context-free language possesses. We have used this to prove that 
languages such as {a n b n c n : n > 0} are not context-free. 

The Church-Turing Thesis: In Chapter 4, we considered “reasonable” 
computational devices that model real computers. Examples of such devices 
are Turing machines (with one or more tapes) and Java programs. It turns 
out that all known “reasonable” devices are equivalent, i.e., can be converted 
to each other. This led to the Church-Turing Thesis: 

• Every computational process that is intuitively considered to be an 
algorithm can be converted to a Turing machine. 
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Decidable and enumerable languages: These classes of languages were 
considered in Chapter 5. They are defined based on “reasonable” computa¬ 
tional devices, such as Turing machines and Java programs. We have seen 
that 

1 . every context-free language is decidable, and 

2 . every decidable language is enumerable. 

Moreover, 

1 . there exist languages, for example {a n b n c n : n > 0}, that are decidable, 
but not context-free, 

2. there exist languages, for example the Halting Problem, that are enu¬ 
merable, but not decidable, 

3. there exist languages, for example the complement of the Halting Prob¬ 
lem, that are not enumerable. 

In fact, 

1 . the class of all languages is not countable, whereas 

2 . the class of all enumerable languages is countable. 

The following statements are equivalent: 

1 . The language A is decidable. 

2. Both A and its complement A are enumerable. 

Complexity classes: These classes of languages were considered in Chap¬ 
ter 6 . 

1. The class P consists of all languages that can be decided in polynomial 
time by a deterministic Turing machine. 

2. The class NP consists of all languages that can be decided in poly¬ 
nomial time by a nondeterministic Turing machine. Equivalently, a 
language A is in the class NP, if for every string w G A, there exists a 
“solution” s, such that (i) the length of s is polynomial in the length 
of w, and (ii) the correctness of s can be verified in polynomial time. 
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The following properties hold: 

1. Every context-free language is in P. (We did not prove this). 

2. Every language in P is also in NP. 

3. It is not known if there exist languages that are in NP, but not in P. 

4. Every language in NP is decidable. 

We have introduced reductions to define the notion of a language B to be 
“at least as hard” as a language A. A language B is called NP-complete, if 

1. B belongs to the class NP, and 

2. B is at least as hard as every language in the class NP. 

We have seen that NP-complete exist. 

The figure below summarizes the relationships among the various classes 
of languages. 
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